{"ID":91415,"post_author":"9412100","post_date":"2020-07-07 16:31:09","post_date_gmt":"2020-07-07 20:31:09","post_content":"","post_title":"LIMSjournal - Spring 2020","post_excerpt":"","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"limsjournal-spring-2020","to_ping":"","pinged":"","post_modified":"2020-07-07 16:43:31","post_modified_gmt":"2020-07-07 20:43:31","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.limsforum.com\/?post_type=ebook&p=91415","menu_order":0,"post_type":"ebook","post_mime_type":"","comment_count":"0","filter":"","_ebook_metadata":{"enabled":"on","private":"0","guid":"A4214AC5-1CCB-4786-822C-CE4F6E85F906","title":"LIMSjournal - Spring 2020","subtitle":"Volume 6, Issue 1","cover_theme":"nico_7","cover_image":"https:\/\/www.limsforum.com\/wp-content\/plugins\/rdp-ebook-builder\/pl\/cover.php?cover_style=nico_7&subtitle=Volume+6%2C+Issue+1&editor=Shawn+Douglas&title=LIMSjournal+-+Spring+2020&title_image=https%3A%2F%2Fs3.limsforum.com%2Fwww.limsforum.com%2Fwp-content%2Fuploads%2FFig1_Mtonga_AfricanJofLabMed2019_8-1.jpg&publisher=LabLynx+Press","editor":"Shawn Douglas","publisher":"LabLynx Press","author_id":"26","image_url":"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/73\/Fig1_Mtonga_AfricanJofLabMed2019_8-1.jpg","items":{"d14ab23a070a6dedd70edd1f31559d55_type":"article","d14ab23a070a6dedd70edd1f31559d55_title":"Leveraging conservation action with open\u2010source hardware (Hill et al. 2019)","d14ab23a070a6dedd70edd1f31559d55_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware","d14ab23a070a6dedd70edd1f31559d55_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Leveraging conservation action with open\u2010source hardware\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nLeveraging conservation action with open\u2010source hardwareJournal\n \nConservation LettersAuthor(s)\n \nHill, Andrew P.; Davies, Alasdair; Prince, Peter; Snaddon, Jake L.; Doncaster, C. Patrick; Rogers, AlexAuthor affiliation(s)\n \nUniversity of Southampton, Zoological Society of London, University of OxfordPrimary contact\n \nEmail: ah1u14 at soton dot ac dot ukYear published\n \n2019Volume and issue\n \n12(5)Page(s)\n \ne12661DOI\n \n10.1111\/conl.12661ISSN\n \n1755-263XDistribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/conbio.onlinelibrary.wiley.com\/doi\/full\/10.1111\/conl.12661Download\n \nhttps:\/\/conbio.onlinelibrary.wiley.com\/doi\/epdf\/10.1111\/conl.12661 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 A framework to support OSH for conservation \n4 Case study \n5 Leveraging frontline action \n6 Acknowledgements \n\n6.1 Author contributions \n6.2 Funding \n6.3 Conflict of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nData collection by conservation biologists is undergoing radical change, with researchers collaborating across disciplines to create bespoke, low\u2010cost monitoring equipment from open\u2010source hardware (OSH). Compared to commercial hardware, OSH dramatically reduces participation costs. Four barriers currently hold back its wide adoption: (1) user inexperience inhibits initial uptake; (2) complex and costly manufacturing\/distribution procedures impede global dissemination; (3) lack of creator support results in lapsed projects; and (4) lack of user support degrades continued utility in the field. Here, we propose a framework to address these barriers, illustrating how OSH offers a route to rapid expansion of community\u2010driven conservation action.\nKeywords: biodiversity monitoring, conservation technology, environmental monitoring, open\u2010source hardware, open\u2010source software\n\nIntroduction \nConservation policy urgently needs accessible, affordable, fit\u2010for\u2010purpose tools to address unprecedented reductions in global biodiversity and rise in illegal wildlife trade (IWT).[1] National governments are now committing to the wild\u2010tech sector, in which open science plays a vital role. For example, the U.K. government is funding initiatives to tackle IWT using innovative open data standards.[2] In this policy perspective, we identify current barriers to the wide adoption of open\u2010source technology and propose a framework for addressing them.\nOver the last 30 years, conservation biology has seen a shift toward data transparency with the growth of open science, including open\u2010access journals, websites hosting open data, software and hardware, and sharing through social media.[3] The new openness has generated a profound change in the ways that hardware and software are developed, leading to conservation biologists collaborating with engineers to create bespoke tools for their specific applications.[4][5] Readily available open\u2010source hardware (OSH) is increasingly used in the rapid and cheap development of deployable prototypes.[6] The fields of conservation, ecology, and environmental sciences have seen an increased uptake in OSH over the last four years, with over a hundred publications reporting on scientific tools created using open\u2010source microcomputers, such as the Raspberry Pi or Arduino.[7][8][9][10][11][12][13]\nThe OSH designation refers to the intellectual property, design principles, and legality of freely available hardware design files, which in their most liberal form can be used to manufacture, distribute, and sell the physically constructed product. Design files consist of circuit\u2010board schematics, circuit\u2010board layout files, and the software source code that together permit construction of a piece of electronic hardware. OSH provides transparency, allowing full public scrutiny of designs to the benefit of their scientific integrity. Open designs create freedom to customize technology for specific applications.[14] The unrestricted access of developers to user needs, and users to developer designs, facilitates rapid community prototyping, either by centrally managed revisions based on user feedback or by user modifications on original designs.[15] The resulting self\u2010made equipment enables replicable data to be gathered at a lower cost than can be achieved with commercial hardware of equivalent utility.[16] These benefits have enabled OSH to colonize niches in technology markets previously unreachable by models based on intellectual property.[17]\nMany barriers still lie in the way of implementing OSH for conservation purposes. Conservation practitioners must assemble the technology manually, generally with no support other than build instructions. Organizations formed around OSH find it difficult to obtain financial resources to continue development[18], often resulting in project termination soon after initial funds are spent.[19] New creators inadvertently reinvent tools when the design files of previously created equipment lapse or become lost. Commercial hardware retains an advantage in this respect with the higher financial outlay paying for product delivery, guarantees, and after\u2010sales care.\nOvercoming current barriers to OSH will require (1) establishing procedures for the manufacture and distribution of hardware that facilitate access and dissemination among the conservation community; (2) financial support for product maintenance; (3) nontechnical instructions for implementing OSH; and (4) after\u2010sale support for continued utility in the field. Models now exist to support adoption of open\u2010source software, such as Canonical providing commercial services for consumers of Linux Ubuntu operating systems. Such frameworks are still lacking, however, to support the not\u2010for\u2010profit uptake and implementation of OSH for conservation. Although profitable businesses are being built around OSH, they meet a demand that comes principally from technically savvy users, capable of building their own hardware from published design files.[20] Conservation practitioners largely fall into a different category of user. They often have limited technical electronics know\u2010how, or they have limited resources for technical training. These users typically require others to build the hardware for them. They remain hard to target for OSH business models due to the complexities that go with hand fabricating hardware from an open design. With appropriate support, however, conservation practitioners are best placed to apply OSH to conservation actions.\nHere, we introduce a provisional framework for developing and sustaining the life cycle of not\u2010for\u2010profit OSH for conservationists. The framework addresses current technical barriers to manufacture and presents simple guidelines for distribution, user accessibility, creator support, and user support. It comprises a set of defined product\u2010development processes that guide a collaborative team through the life cycle of an open\u2010source product, from construction and after\u2010sales support, to the reinvestment strategy that sustains the creators and community. We demonstrate an application of the framework with a real\u2010world case study of an OSH product in the form of an acoustic monitoring device.[21] The case study serves to illustrate how the framework unlocks useful technology for local communities, researchers funded by government research councils, and individuals funded by non\u2010government organizations (NGOs). In recent years, similar frameworks, such as Crowd Supply, have been shown to increase the adoption of proprietary products by consumers[22] and even to improve competitive advantage through crowdsourced tools.[23] We argue for wide adoption of flexible approaches of this sort by conservation NGOs and universities in particular. A framework can facilitate rapid uptake of OSH for conservation activities supported by these organizations, thereby fostering the proliferation of local\u2010scale projects that lead to global\u2010scale action.[24][25]\n\nA framework to support OSH for conservation \nThe framework has six phases, each with formulated guidelines: (1) hardware, (2) manufacture, (3) release, (4) distribution, (5) support, and (6) sustainability. (See Figure 1.) The framework is formed around a management team consisting of a creator group (engineering team), which leads the technical developments of the technology, and a logistics group (funder\/NGO\/university), which leads logistical and financial operations. This unified entity requires an open and collaborative relationship between the two groups, with clear lines of responsibility (detailed in Table 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. An overview of the framework for developing, funding, supporting, and sustaining open\u2010source hardware (OSH) for conservation. The framework differs from a traditional business model for a commercial product in having open\u2010source licenses, relying on word\u2010of\u2010mouth in lieu of an advertising budget, allowing group\u2010purchase only, explicitly excluding housing for electronic components and guarantees on product performance, and outsourcing after\u2010sales care to web forums.\n\n\n\n\n\n\n\n\n\nTable 1. Distribution of responsibilities for the management team, between the creator group, which focuses on hardware issues, and the logistics group, which focuses on assurance issues\n\n\nManagement team group\n\nResponsibility\n\n\nCreator group\n\n\nConception of idea\r\n\nCircuit design\r\n\nHardware development\r\n\nSoftware development\u2014website, firmware, supporting software and algorithm design\r\n\nPrototyping\r\n\nBench testing\r\n\nField testing\r\n\nChoosing appropriate manufacturers\r\n\nTesting small\u2010batch build quality of chosen manufacturer\r\n\nSourcing components for crowd funding order\r\n\nTesting alternative components if original components become obsolete\r\n\nWebsite maintenance\r\n\nTechnical community support via website forum\n\n\n\nLogistics group\n\n\nnitiating crowd funded campaign\r\n\nTaking payment from crowdfunding organisation\r\n\nCo\u2010organising distribution of units\r\n\nAssurance\u2010related community support via crowd funders website\r\n\nHolding pool of funds\r\n\nDistributing pool of funds to support technology development\n\n\n\nBoth groups\n\nSocial media presence\n\n\n\nThe end users are the communities requiring the conservation hardware, such as local conservation activists, research scientists, educators, and wildlife enthusiasts. The framework provides a provisional set of management guidelines for OSH development, and a scalable method for the community to acquire and use the created technology. The framework generates a pool of crowdsourced funds out of unit sales margins obtained in the bulk manufacturing of hardware. Funds can be reinvested back into the hardware project to sustain its lifespan. In addition to providing support for creators and users, the framework minimizes the investment in human and financial capacity needed to initiate, manufacture, and distribute OSH. It makes use of new websites for crowdfunding and for turn\u2010key electronics manufacturing to provide tailored and cheap solutions for nontechnical conservation organizations. A framework can be viewed as successful if it is able to harness a large community of individuals to acquire low\u2010cost single units from a high-volume manufacturing process, while also creating extra funds to continue support. Success will depend on simple hardware construction to reduce overhead when manufacturing at bulk, and the timely orchestration of a large group of buyers to crowdfund industrial manufacture of the hardware.[26]\nThe initial \u201chardware\u201d phase is the design stage, which commences after the creator group has proven the feasibility of using a particular technology for a desired conservation task. The proven hardware needs to be adapted to an open\u2010source design, and initial investment is required to fund this development. To open source a technology means to apply an open license to it. Various licenses are commonly used to define how OSH design files can be adapted, shared, or commercialized by the user community. (See Table 2.) Each license has its own benefits, depending on the goal. For example, the most open license, CC0, may be useful for community ownership of the design, thus allowing it to be shared, adapted, and commercialized without the need to attribute changes to the original creator(s). Any of the OSH licenses can be used in the framework. Oberloier and Pearce[27] detail formal procedures for designers of OSH.\n\n\n\n\n\n\n\nTable 2. Comparison of the available Creative Commons OSH licenses\n\n\nCreative Commons licenses\n\nAdaptations can be shared\n\nCommercial use\n\n\nCC0 (No attribution)\n\nYes\n\nYes\n\n\nAttribution (BY)\n\nYes\n\nYes\n\n\nShareAlike (CC BY\u2010SA)\n\nYes, but must share alike\n\nYes\n\n\nNonCommercial (CC BY\u2010NC)\n\nYes\n\nNo\n\n\nNonCommercial\u2010NoDerivatives (CC BY\u2010NC\u2010ND)\n\nNo\n\nNo\n\n\nNoDerivatives (BY\u2010ND)\n\nNo\n\nYes\n\n\nNonCommercial\u2010ShareAlike (CC BY\u2010NC\u2010SA)\n\nYes, but must share alike\n\nNo\n\n\n\nThe \u201cmanufacture\u201d phase involves physical fabrication and testing of the subsequent OSH design. Manufacturing often involves complex fabrication processes, including sourcing parts, assembly of printed circuit boards (PCBs), programming the PCBs, and developing housing for the product suitable for field deployment. Keeping this sequence as simple as possible minimizes manufacturing costs. Web services chosen for PCB manufacture should be trialed by testing the build quality of small batches of the developed hardware units before release. Tests should involve checking the quality of assembled PCBs, assessing the fit\u2010for\u2010purpose functionality of the final product, and providing solutions to assembly problems found during manufacture. Issues found at this stage need to be resolved before continuing to the next phase. Spare devices built during this process can be used as trial devices or backup units to support after\u2010sales assurance issues.\nThe \u201crelease\u201d phase consists of the publication, promotion, sales, and large\u2010batch manufacture of the hardware units. Technology design files are first uploaded to a publicly accessible open\u2010source hosting website to allow community access. The guiding principle of this phase is to minimize the cost per unit for users while also generating a pool of funds. Large\u2010scale industrial manufacturing processes reduce unit material cost and assembly time; however, they require an initial sum of money to bulk order before making any sales. To this end, minimum quantity batch manufacturing can be crowdsourced to reduce risk and produce an economy of scale.[28] A crowdfunding platform is required to collect user's funds during funding campaigns. Campaigns should run at regular intervals, and each should end when a maximum order quantity is reached, or a time period is elapsed. The crowdfunding campaign should be publicized on social media and conservation forums, so as to gain traction and reduce the risk of failing to meet the monetary target.\nThe \u201cdistribution\u201d phase consists of the global dissemination of the manufactured units. Depending on the crowdfunding organization used, the logistics group may need to take on the role of global distributor, or employ a dedicated distribution house to ship and track deliveries.\nThe \u201csupport\u201d phase provides technical and logistical assistance to the community of users. Users may require support with diverse issues, including devices getting lost in transit, out\u2010of\u2010box malfunctions, and difficulties with operating the device. A mechanism should be established for users to send their end\u2010of\u2010life devices back to the creator group for re\u2010use or disposal according to the Waste Electrical and Electronic Equipment (WEEE) Regulations.\n\u201cSustainability\u201d encompasses the reinvestment strategy to facilitate continued functioning of the technology into the future. It involves addressing how pooled funds can be generated and used to continue future development and promote future purchases. To create pooled funds, a margin should be added to each unit sold that generates enough funds from the minimum quantity order to sustain the team until the next group purchase campaign ends. The crowdfunding campaign should set its goal to this minimum batch order. Costs incurred by the logistics group to manage logistical operations, such as employing a part\u2010time member of staff and purchasing packaging for shipping, should be reimbursed from this generated fund. The creator and logistics groups should take a percentage of funds to support their tasks after each group funding campaign. The remaining pool of funds can be held by the logistics group and made accessible to the creator group as needed to pay for further support, bug fixes, and further enhancement or development.\n\nCase study \nThe framework was developed and formalized for the open\u2010source manufacture and distribution of an acoustic monitoring device called AudioMoth, created by Open Acoustic Devices (OAD). Within the management team, the creator group comprised OAD, and the logistics group comprised a small conservation NGO called the Arribada Initiative.\nThe AudioMoth case study used the Creative Commons Attribution 4.0 International (CC BY) open\u2010source license. (See Table 2.) This attribution allowed full participation by the community in all aspects of design, attributing all changes to the original creators. The license allowed other manufacturers to construct and re\u2010sell the platform independently (e.g., LabMaker). This brought mixed benefits: it raised awareness and distribution of devices, while also reducing the funds available for sustaining the devices, although in this case only by about one\u2010tenth.\nHardware was designed to fit a single fixed part, consisting of one single\u2010sided PCB that could be purchased from a single manufacturer. The creator group developed the hardware as part of their research budget from U.K. Research Council grants. Development of the product was funded by grants totaling \u223cUSD $13,000 per\/year over three years for two PhD students, excluding stipends and course fees.\nThe chosen manufacturer was CircuitHub. This online web service specializes in assembly of open\u2010source PCBs. It enables design files to be uploaded, shared, manufactured, and delivered to order, with transparent batch size, lead time, and pricing. It first built small test batches of prototype PCBs, which were used to evaluate the single\u2010board construction and address any build issues. Test boards had a 20\u2010day lead time and cost USD $60 per unit for a batch of 50.\nThe finalized AudioMoth design files were published on the open\u2010source hosting websites GitHub and CircuitHub. This made them freely available to adapt and share. CircuitHub's online quotation tool was used to calculate the unit selling price on a manufactured batch of 200 units. A further 20% was added to this unit price, which was calculated to generate enough funds to support global distribution, after\u2010sales support, and future enhancements. Cost\u2010free promotion of AudioMoth was achieved through the WILDLABS forum, social media, and the OAD subscription page. Trial devices were distributed to potential users to proliferate the spread of information throughout the conservation community and to generate feedback to the creator group on the AudioMoth utility.\nThe crowdfunding organization GroupGets coordinated the group purchase campaign for each batch order from CircuitHub. GroupGets linked back to OAD's subscription page to help the creators gauge the level of online interest. To fulfill a cost\u2010effective manufacturing run, each campaign had to gain a minimum of 200 backers.\nGroupGets handled postage to users based in the United States. For orders bound elsewhere, GroupGets batch posted the ordered units to the U.K. headquarters of the logistics group. This minimized individual customs declarations, freeing up sufficient funds from the price margin for the logistics group to distribute units to the rest of the world. The logistics group handled all non\u2010U.S. distribution until the rising level of demand necessitated a dedicated distribution house. Weengs was then selected to receive devices directly from GroupGets and manage the major distribution channels to the U.K. and E.U.\nThe creator group maintained a website supporting users of AudioMoth. This was updated on a weekly basis, taking approximately two person\u2010hours\/week. The website hosted usage instructions, open\u2010source design files, and a support forum. The forum enabled users to discuss ongoing projects, report technical issues, and suggest modifications to the open\u2010source design files. Over time, it became partially self\u2010sustaining as the community of users gained expertise in the technology. The creator group was still required to provide solutions to more technical issues.\nThe case study supported six rounds of group\u2010purchase campaigns over 18 months, from October 2017 to March 2019. These campaigns generated a pool of funds to finance U.S. distribution, hardware improvements, ongoing user support for all purchased devices, and community\u2010based events, including workshops for users. The sixth group\u2010funding campaign introduced the first hardware modification to the design files. A lesson learned during this change was the importance of maintaining backwards compatible firmware across hardware versions.\nThe six campaigns resulted in 5,242 unit sales in total, with 34% purchased by universities, 23% by conservation organizations, 11% by businesses\/consultants, 1% by government agencies, and 30% by unspecified individuals. All devices sold for USD $49.99 per unit, creating a net revenue of USD $262,048 and a pool of funds after device manufacture and distribution of USD $58,188 (discounting the initial external investment). Pooled funds were managed by the logistics group throughout the study. The remaining pool will continue to be reinvested back into the framework to pay for ongoing maintenance of the website, to support the community of users, and to further improve the hardware. (See Table 3.)\n\n\n\n\n\n\n\nTable 3. Breakdown of revenue and expenditure throughout the 18\u2010month AudioMoth case study using the framework to support OSH\n\n\nPayments in\/out\n\nAmount (USD)\n\n\nSale of 5,242 units\n\n$262,047.58\n\n\nManufacturing costs\n\n\u2212$141,404.28\n\n\nGroup purchase fees\n\n\u2212$36,871.01\n\n\nPostage fees\n\n\u2212$11,228.25\n\n\nCommunity activity costs\n\n\u2212$8,076.19\n\n\nAdditional hardware improvements\n\n\u2212$6,279.67\n\n\nTotal pool of funds\n\n$58,188.18\n\n\n\nLeveraging frontline action \nThe case study demonstrates the capability of the framework for sustaining the lifecycle of an open\u2010source conservation tool, while also stimulating global deployment of conservation technology. The framework has to date enabled 687 separate projects to purchase OSH. Purchasers are globally distributed (Figure 2a), with clusters around the financial hubs of advanced economies, particularly in Europe (63%), North America (23%), and Australia (8%). Purchases made elsewhere include Central and South America (4%), Asia (2%), and Africa (<1%).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. The global impact of the framework to support OSH: (a) Heat map of all OSH purchase locations; (b) deployment locations and field applications of a 14% sample of purchasers\n\n\n\nFourteen percent of the framework's end\u2010users posted on social media about conservation work using AudioMoths, with conservation action occurring largely away from purchase locations. For example, Africa was the source country for only two sales, but the deployment location for 23% of purchases. Devices were used for a wide range of applications, including terrestrial monitoring of individual species, general ecosystem soundscape analysis, monitoring of human\u2013wildlife conflicts, experimental marine surveys, and university\u2010level education (Figure 2b). Also by lowering the cost of participating in applied acoustic monitoring research, the application of the framework to AudioMoth facilitated public engagement, including citizen science projects for a national bat survey[29], biodiversity mapping surveys[30][31][32], and engagement by the general public in wildlife monitoring.[33] It also led to the development of new methods to address previously untestable research questions.[34]\nThose participating in group purchases of AudioMoths actively shared modifications and enhancements, including designs for housing the devices, on independent forum pages as well as on the OAD online community forum where changes could be validated by the creator group. Although the framework functions well for AudioMoth, more complicated OSH products may need additional manufacturing stages. Incorporating a framework to support OSH into NGO policy still requires substantial work, including further case studies of different conservation monitoring equipment. Without a framework of the type we propose, however, existing OSH will continue to have short life spans, and remain out of reach for the majority of conservation biologists.\nCreating legacy value for OSH and proprietary projects used for conservation should be the first principle of action for conservation organizations and universities. Online community platforms for conservation technology such as WILDLABS could take a lead in policy adoption, creating an official platform for recording long\u2010term OSH findings for the community to share and learn from. It is important to push access to OSH toward communities outside of the pockets of wealth and high opportunity that the framework may initially serve. Future developments should investigate benefits and costs of completely transitioning the framework to the conservation community, ways of making it more accessible to those communities typically underserved by citizen science, and methods of community\u2010led support for conservation tools.\n\nAcknowledgements \nWe thank Jos\u00e9 Lahoz\u2010Monfort for organizing the ARC CEED workshop entitled \u201cStepping into the future: Driving the Technology Agenda in Ecology and Conservation,\u201d where the OAD and Arribada Initiative seeded the idea for a framework. We thank Stephanie O'Donnell for help with establishing the user\u2010group and providing data for Figure 2b, Anne Dangerfield for logistics management, and Holly Baines for user support. A special thanks goes to the AudioMoth community.\n\nAuthor contributions \nAndrew P. Hill, Peter Prince, Alex Rogers, C. Patrick Doncaster, and Jake L. Snaddon contributed to the design of AudioMoth acoustic sensors, with Alex Rogers, Andrew P. Hill, and Peter Prince leading hardware and software development; Alasdair Davies led the logistics group and AudioMoth distribution service; and all authors contributed to writing of the paper.\n\nFunding \nThis work was supported by a U.K. EPSRC Studentship (1658469) to A. Hill, and a U.K. NERC SPITFIRE DTP award (NE\/L002531\/1) to P. Prince. \n\nConflict of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 The Royal Society (08 October 2018). \"Illegal wildlife trade\". https:\/\/royalsociety.org\/topics-policy\/projects\/illegal-wildlife-trade\/ .   \n\n\u2191 U.K. Government (31 January 2019). \"Digital revolution to use the power of data to combat illegal wildlife trade and reduce food waste\". https:\/\/www.gov.uk\/government\/news\/digital-revolution-to-use-the-power-of-data-to-combat-illegal-wildlife-trade-and-reduce-food-waste .   \n\n\u2191 Hampton, S.E.; Anderson, S.S.; Bagby, S.C. et al. (2015). \"The Tao of open science for ecology\". Ecosphere 6 (7): 1\u201313. doi:10.1890\/ES14-00402.1.   \n\n\u2191 Berger-Tal, O.; Lahoz-Monfort, J.J. (2018). \"Conservation technology: The next generation\". Conservation Letters 11 (6): e12458. doi:10.1111\/conl.12458.   \n\n\u2191 Kwok, R. (2017). \"Field Instruments: Build it yourself\". Nature 545: 253\u201355. doi:10.1038\/nj7653-253a.   \n\n\u2191 Pearce, J.M. (2014). \"Laboratory equipment: Cut costs with open-source hardware\". Nature 505 (7485): 618. doi:10.1038\/505618d. PMID 24476879.   \n\n\u2191 Ahmad, A.; Nadzri, M.M.M.; Rosli, I.M. et al. (2018). \"Rapid Prototyping of Wireless Image Transmission for Wildlife (Tiger) Monitoring System - A Preliminary Study\". Journal of Telecommunication, Electronic and Computer Engineering 10 (2-5): 75\u201379. http:\/\/journal.utem.edu.my\/index.php\/jtec\/article\/view\/4354 .   \n\n\u2191 Nazir, S.; Newey, S.; Irvine, R.J. et al. (2017). \"WiseEye: Next Generation Expandable and Programmable Camera Trap Platform for Wildlife Research\". PLoS One 12 (1): e0169758. doi:10.1371\/journal.pone.0169758. PMC PMC5226779. PMID 28076444. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5226779 .   \n\n\u2191 Sankupellay, M.; McLaughlin, C.; Davis, J. et al. (2016). \"B4 - Brisbane Backyard Bird Box: Connecting People to the Environment\". Proceedings of the 2016 ACM Conference Companion Publication on Designing Interactive Systems: 9\u201312. doi:10.1145\/2908805.2908808.   \n\n\u2191 Soro, A.; Brereton, M.; Dema, T. et al. (2018). \"The Ambient Birdhouse: An IoT Device to Discover Birds and Engage with Nature\". Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems: 1\u201313. doi:10.1145\/3173574.3173971.   \n\n\u2191 Tan, T.F.; Teoh, S.S.; Fow, J.E. et al. (2016). \"Embedded human detection system based on thermal and infrared sensors for anti-poaching application\". Proceedings of the 2016 IEEE Conference on Systems, Process and Control: 37\u201342. doi:10.1109\/SPC.2016.7920700.   \n\n\u2191 Maina, C.W.; Muchiri, D.; Njoroge, P. (2016). \"A Bioacoustic Record of a Conservancy in the Mount Kenya Ecosystem\". Biodiversity Data Journal 4: e9906. doi:10.3897\/BDJ.4.e9906.   \n\n\u2191 Whytock, R.C.; Christie, J. (2017). \"Solo: An open source, customizable and inexpensive audio recorder for bioacoustic research\". Methods in Ecology and Evolution 8 (3): 308\u201312. doi:10.1111\/2041-210X.12678.   \n\n\u2191 Kling, J. (2018). \"DIY goes in vivo\". Lab Animal 47: 143\u201346. doi:10.1038\/s41684-018-0066-z.   \n\n\u2191 O'Mahony, S. (2007). \"The governance of open source initiatives: what does it mean to be community managed?\". Journal of Management and Governance 11 (2): 139\u201350. doi:10.1007\/s10997-007-9024-7.   \n\n\u2191 Drack, M.; Hartmann, F.; Bauer, S. et al. (2018). \"The importance of open and frugal labware\". Nature Electronics 1 (9): 484\u201386. doi:10.1038\/s41928-018-0133-x.   \n\n\u2191 Hsing, P.-Y. (2018). \"Sustainable Innovation for Open Hardware and Open Science \u2013 Lessons from The Hardware Hacker\". Journal of Open Hardware 2 (1): 4. doi:10.5334\/joh.11.   \n\n\u2191 Li, Z.; Seering, W.; Wallace, D. (2018). \"Understanding Value Propositions and Revenue Models in Open Source Hardware Companies\". Proceedings from the 2018 International Conference on Innovation and Entrepreneurship: 214\u201323.   \n\n\u2191 Iacona, G.; Ramachandra, A.; McGowan, J. et al. (2019). \"Identifying technology solutions to bring conservation into the innovation era\". Frontiers in Ecology and the Environment 17 (10): 591\u201398. doi:10.1002\/fee.2111.   \n\n\u2191 Pearce, J.M. (2017). \"Emerging Business Models for Open Source Hardware\". Journal of Open Hardware 1 (1): 2. doi:10.5334\/joh.4.   \n\n\u2191 Hill, A.P.; Prince, P.; Covarrubias, E.P. et al. (2017). \"AudioMoth: Evaluation of a smart open acoustic device for monitoring biodiversity and the environment\". Methods in Ecology and Evolution 9 (5): 1199\u20131211. doi:10.1111\/2041-210X.12955.   \n\n\u2191 Ilin, P.; Platis, D. Hammouda, I. (2018). \"Insights into the Trilateral Relationship of Crowdfunding Campaigns, Open Source and Communities\". Proceedings from the 2018 Open Source Systems: Enterprise Software and Solutions: 26\u201336. doi:10.1007\/978-3-319-92375-8_3.   \n\n\u2191 Nagle, F. (2018). \"Learning by Contributing: Gaining Competitive Advantage Through Contribution to Crowdsourced Public Goods\". Organization Science 29 (4): 547\u2013753. doi:10.1287\/orsc.2018.1202.   \n\n\u2191 Arlettaz, R.; Schaub, M.; Fournier, J. et al. (2010). \"From Publications to Public Actions: When Conservation Biologists Bridge the Gap between Research and Implementation\". BioScience 60 (10): 835\u201342. doi:10.1525\/bio.2010.60.10.10.   \n\n\u2191 Pocock, M.J.O.; Chandler, M.; Bonney, R. et al. (2018). \"A Vision for Global Biodiversity Monitoring With Citizen Science\". Advances in Ecological Research 59: 169\u2013223. doi:10.1016\/bs.aecr.2018.06.003.   \n\n\u2191 Kohler, T.; Chesbrough, H. (2019). \"From collaborative community to competitive market: The quest to build a crowdsourcing platform for social innovation\". R&D Management 49 (3): 356\u201368. doi:10.1111\/radm.12372.   \n\n\u2191 Oberloier, S.; Pearce, J.M. (2017). \"General Design Procedure for Free and Open-Source Hardware for Scientific Equipment\". Designs 2 (1): 2. doi:10.3390\/designs2010002.   \n\n\u2191 Wheat, R.E.; Wang, Y.; Byrnes, J.E.; Ranganathan, J. (2013). \"Raising money for scientific research through crowdfunding\". Trends in Ecology & Evolution 28 (2): 71\u20132. doi:10.1016\/j.tree.2012.11.001.   \n\n\u2191 Bat Conservation Trust (2019). \"National Bat Monitoring Programme\". https:\/\/www.bats.org.uk\/our-work\/national-bat-monitoring-programme .   \n\n\u2191 Point Blue Conservation Science (2019). \"Soundscapes to Landscapes\". https:\/\/www.soundscapes2landscapes.org .   \n\n\u2191 Bodon, S. (26 February 2019). \"Pitt Researchers Are Eavesdropping On Birds In The Name Of Science\". 90.5 WESA. https:\/\/www.wesa.fm\/post\/pitt-researchers-are-eavesdropping-birds-name-science .   \n\n\u2191 Cetalingua (2019). \"Participatory Platform for Citizen Scientists\". https:\/\/www.cetalingua.com\/citizen-science\/ .   \n\n\u2191 Gillings, S.; Moran, N. (2019). \"AudioMoth\". NOCMIG: Tips for Recording Nocturnal Bird Migration. https:\/\/www.nocmig.com\/audiomoth .   \n\n\u2191 Pi\u00f1a\u2010Covarrubias, E.; Hill, A.P.; Prince, P. et al. (2019). \"Optimization of sensor deployment for acoustic detection and localization in terrestrial environments\". Remote Sensing in Ecology and Conservation 5 (2): 180\u201392. doi:10.1002\/rse2.97.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance. The Iacona et al. in-press article was updated with the now-published version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\">https:\/\/www.limswiki.org\/index.php\/Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on biodiversity informaticsLIMSwiki journal articles on information technologyLIMSwiki journal articles on open source\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 4 February 2020, at 17:58.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 208 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","d14ab23a070a6dedd70edd1f31559d55_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Leveraging_conservation_action_with_open\u2010source_hardware skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Leveraging conservation action with open\u2010source hardware<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Data collection by conservation biologists is undergoing radical change, with researchers collaborating across disciplines to create bespoke, low\u2010cost <a href=\"https:\/\/www.limswiki.org\/index.php\/Environmental_monitoring\" title=\"Environmental monitoring\" class=\"wiki-link\" data-key=\"4671d3cbb4201e1698d0fb44369dc1ea\">monitoring<\/a> equipment from open\u2010source hardware (OSH). Compared to commercial hardware, OSH dramatically reduces participation costs. Four barriers currently hold back its wide adoption: (1) user inexperience inhibits initial uptake; (2) complex and costly manufacturing\/distribution procedures impede global dissemination; (3) lack of creator support results in lapsed projects; and (4) lack of user support degrades continued utility in the field. Here, we propose a framework to address these barriers, illustrating how OSH offers a route to rapid expansion of community\u2010driven conservation action.\n<\/p><p><b>Keywords<\/b>: biodiversity monitoring, conservation technology, environmental monitoring, open\u2010source hardware, open\u2010source software\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Conservation policy urgently needs accessible, affordable, fit\u2010for\u2010purpose tools to address unprecedented reductions in global biodiversity and rise in illegal wildlife trade (IWT).<sup id=\"rdp-ebb-cite_ref-RSIllegal18_1-0\" class=\"reference\"><a href=\"#cite_note-RSIllegal18-1\">[1]<\/a><\/sup> National governments are now committing to the wild\u2010tech sector, in which open science plays a vital role. For example, the U.K. government is funding initiatives to tackle IWT using innovative open data standards.<sup id=\"rdp-ebb-cite_ref-UKGovDigital19_2-0\" class=\"reference\"><a href=\"#cite_note-UKGovDigital19-2\">[2]<\/a><\/sup> In this policy perspective, we identify current barriers to the wide adoption of open\u2010source technology and propose a framework for addressing them.\n<\/p><p>Over the last 30 years, conservation biology has seen a shift toward data transparency with the growth of open science, including open\u2010access journals, websites hosting open data, <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software\" title=\"Free and open-source software\" class=\"wiki-link\" data-key=\"688dbf2b7fb58c69c6ad476828ee2c6a\">software<\/a> and hardware, and sharing through social media.<sup id=\"rdp-ebb-cite_ref-HamptonTheTao15_3-0\" class=\"reference\"><a href=\"#cite_note-HamptonTheTao15-3\">[3]<\/a><\/sup> The new openness has generated a profound change in the ways that hardware and software are developed, leading to conservation biologists collaborating with engineers to create bespoke tools for their specific applications.<sup id=\"rdp-ebb-cite_ref-Berger-TalConserv18_4-0\" class=\"reference\"><a href=\"#cite_note-Berger-TalConserv18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KwokField17_5-0\" class=\"reference\"><a href=\"#cite_note-KwokField17-5\">[5]<\/a><\/sup> Readily available open\u2010source hardware (OSH) is increasingly used in the rapid and cheap development of deployable prototypes.<sup id=\"rdp-ebb-cite_ref-PearceLab14_6-0\" class=\"reference\"><a href=\"#cite_note-PearceLab14-6\">[6]<\/a><\/sup> The fields of conservation, ecology, and environmental sciences have seen an increased uptake in OSH over the last four years, with over a hundred publications reporting on scientific tools created using open\u2010source microcomputers, such as the Raspberry Pi or Arduino.<sup id=\"rdp-ebb-cite_ref-AhmadRapid18_7-0\" class=\"reference\"><a href=\"#cite_note-AhmadRapid18-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NazirWise17_8-0\" class=\"reference\"><a href=\"#cite_note-NazirWise17-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SankupellayB4_16_9-0\" class=\"reference\"><a href=\"#cite_note-SankupellayB4_16-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SoroTheAmbient18_10-0\" class=\"reference\"><a href=\"#cite_note-SoroTheAmbient18-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TanEmbed16_11-0\" class=\"reference\"><a href=\"#cite_note-TanEmbed16-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MainaABio16_12-0\" class=\"reference\"><a href=\"#cite_note-MainaABio16-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WhytockSolo16_13-0\" class=\"reference\"><a href=\"#cite_note-WhytockSolo16-13\">[13]<\/a><\/sup>\n<\/p><p>The OSH designation refers to the intellectual property, design principles, and legality of freely available hardware design files, which in their most liberal form can be used to manufacture, distribute, and sell the physically constructed product. Design files consist of circuit\u2010board schematics, circuit\u2010board layout files, and the software source code that together permit construction of a piece of electronic hardware. OSH provides transparency, allowing full public scrutiny of designs to the benefit of their scientific integrity. Open designs create freedom to customize technology for specific applications.<sup id=\"rdp-ebb-cite_ref-KlingDIY18_14-0\" class=\"reference\"><a href=\"#cite_note-KlingDIY18-14\">[14]<\/a><\/sup> The unrestricted access of developers to user needs, and users to developer designs, facilitates rapid community prototyping, either by centrally managed revisions based on user feedback or by user modifications on original designs.<sup id=\"rdp-ebb-cite_ref-OMahonyTheGov07_15-0\" class=\"reference\"><a href=\"#cite_note-OMahonyTheGov07-15\">[15]<\/a><\/sup> The resulting self\u2010made equipment enables replicable data to be gathered at a lower cost than can be achieved with commercial hardware of equivalent utility.<sup id=\"rdp-ebb-cite_ref-DrackTheImport18_16-0\" class=\"reference\"><a href=\"#cite_note-DrackTheImport18-16\">[16]<\/a><\/sup> These benefits have enabled OSH to colonize niches in technology markets previously unreachable by models based on intellectual property.<sup id=\"rdp-ebb-cite_ref-HsingSustain18_17-0\" class=\"reference\"><a href=\"#cite_note-HsingSustain18-17\">[17]<\/a><\/sup>\n<\/p><p>Many barriers still lie in the way of implementing OSH for conservation purposes. Conservation practitioners must assemble the technology manually, generally with no support other than build instructions. Organizations formed around OSH find it difficult to obtain financial resources to continue development<sup id=\"rdp-ebb-cite_ref-LiUnder18_18-0\" class=\"reference\"><a href=\"#cite_note-LiUnder18-18\">[18]<\/a><\/sup>, often resulting in project termination soon after initial funds are spent.<sup id=\"rdp-ebb-cite_ref-IaconaIdent19_19-0\" class=\"reference\"><a href=\"#cite_note-IaconaIdent19-19\">[19]<\/a><\/sup> New creators inadvertently reinvent tools when the design files of previously created equipment lapse or become lost. Commercial hardware retains an advantage in this respect with the higher financial outlay paying for product delivery, guarantees, and after\u2010sales care.\n<\/p><p>Overcoming current barriers to OSH will require (1) establishing procedures for the manufacture and distribution of hardware that facilitate access and dissemination among the conservation community; (2) financial support for product maintenance; (3) nontechnical instructions for implementing OSH; and (4) after\u2010sale support for continued utility in the field. Models now exist to support adoption of open\u2010source software, such as <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/canonical.com\/\" target=\"_blank\">Canonical<\/a> providing commercial services for consumers of Linux Ubuntu operating systems. Such frameworks are still lacking, however, to support the not\u2010for\u2010profit uptake and implementation of OSH for conservation. Although profitable businesses are being built around OSH, they meet a demand that comes principally from technically savvy users, capable of building their own hardware from published design files.<sup id=\"rdp-ebb-cite_ref-PearceEmerging17_20-0\" class=\"reference\"><a href=\"#cite_note-PearceEmerging17-20\">[20]<\/a><\/sup> Conservation practitioners largely fall into a different category of user. They often have limited technical electronics know\u2010how, or they have limited resources for technical training. These users typically require others to build the hardware for them. They remain hard to target for OSH business models due to the complexities that go with hand fabricating hardware from an open design. With appropriate support, however, conservation practitioners are best placed to apply OSH to conservation actions.\n<\/p><p>Here, we introduce a provisional framework for developing and sustaining the life cycle of not\u2010for\u2010profit OSH for conservationists. The framework addresses current technical barriers to manufacture and presents simple guidelines for distribution, user accessibility, creator support, and user support. It comprises a set of defined product\u2010development processes that guide a collaborative team through the life cycle of an open\u2010source product, from construction and after\u2010sales support, to the reinvestment strategy that sustains the creators and community. We demonstrate an application of the framework with a real\u2010world case study of an OSH product in the form of an acoustic monitoring device.<sup id=\"rdp-ebb-cite_ref-HillAudio17_21-0\" class=\"reference\"><a href=\"#cite_note-HillAudio17-21\">[21]<\/a><\/sup> The case study serves to illustrate how the framework unlocks useful technology for local communities, researchers funded by government research councils, and individuals funded by non\u2010government organizations (NGOs). In recent years, similar frameworks, such as Crowd Supply, have been shown to increase the adoption of proprietary products by consumers<sup id=\"rdp-ebb-cite_ref-IlinInsights18_22-0\" class=\"reference\"><a href=\"#cite_note-IlinInsights18-22\">[22]<\/a><\/sup> and even to improve competitive advantage through crowdsourced tools.<sup id=\"rdp-ebb-cite_ref-NagleLearning18_23-0\" class=\"reference\"><a href=\"#cite_note-NagleLearning18-23\">[23]<\/a><\/sup> We argue for wide adoption of flexible approaches of this sort by conservation NGOs and universities in particular. A framework can facilitate rapid uptake of OSH for conservation activities supported by these organizations, thereby fostering the proliferation of local\u2010scale projects that lead to global\u2010scale action.<sup id=\"rdp-ebb-cite_ref-ArlettazFromPub10_24-0\" class=\"reference\"><a href=\"#cite_note-ArlettazFromPub10-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PocockAVision18_25-0\" class=\"reference\"><a href=\"#cite_note-PocockAVision18-25\">[25]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"A_framework_to_support_OSH_for_conservation\">A framework to support OSH for conservation<\/span><\/h2>\n<p>The framework has six phases, each with formulated guidelines: (1) hardware, (2) manufacture, (3) release, (4) distribution, (5) support, and (6) sustainability. (See Figure 1.) The framework is formed around a management team consisting of a creator group (engineering team), which leads the technical developments of the technology, and a logistics group (funder\/NGO\/university), which leads logistical and financial operations. This unified entity requires an open and collaborative relationship between the two groups, with clear lines of responsibility (detailed in Table 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Hill_ConsLetters2019_12-5.jpg\" class=\"image wiki-link\" data-key=\"34021e7bf58f6e7fe430b9f44e5ac812\"><img alt=\"Fig1 Hill ConsLetters2019 12-5.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ea\/Fig1_Hill_ConsLetters2019_12-5.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> An overview of the framework for developing, funding, supporting, and sustaining open\u2010source hardware (OSH) for conservation. The framework differs from a traditional business model for a commercial product in having open\u2010source licenses, relying on word\u2010of\u2010mouth in lieu of an advertising budget, allowing group\u2010purchase only, explicitly excluding housing for electronic components and guarantees on product performance, and outsourcing after\u2010sales care to web forums.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> Distribution of responsibilities for the management team, between the creator group, which focuses on hardware issues, and the logistics group, which focuses on assurance issues\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Management team group\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Responsibility\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Creator group\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>Conception of idea<br \/>\nCircuit design<br \/>\nHardware development<br \/>\nSoftware development\u2014website, firmware, supporting software and algorithm design<br \/>\nPrototyping<br \/>\nBench testing<br \/>\nField testing<br \/>\nChoosing appropriate manufacturers<br \/>\nTesting small\u2010batch build quality of chosen manufacturer<br \/>\nSourcing components for crowd funding order<br \/>\nTesting alternative components if original components become obsolete<br \/>\nWebsite maintenance<br \/>\nTechnical community support via website forum\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Logistics group\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>nitiating crowd funded campaign<br \/>\nTaking payment from crowdfunding organisation<br \/>\nCo\u2010organising distribution of units<br \/>\nAssurance\u2010related community support via crowd funders website<br \/>\nHolding pool of funds<br \/>\nDistributing pool of funds to support technology development\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Both groups\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Social media presence\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The end users are the communities requiring the conservation hardware, such as local conservation activists, research scientists, educators, and wildlife enthusiasts. The framework provides a provisional set of management guidelines for OSH development, and a scalable method for the community to acquire and use the created technology. The framework generates a pool of crowdsourced funds out of unit sales margins obtained in the bulk manufacturing of hardware. Funds can be reinvested back into the hardware project to sustain its lifespan. In addition to providing support for creators and users, the framework minimizes the investment in human and financial capacity needed to initiate, manufacture, and distribute OSH. It makes use of new websites for crowdfunding and for turn\u2010key electronics manufacturing to provide tailored and cheap solutions for nontechnical conservation organizations. A framework can be viewed as successful if it is able to harness a large community of individuals to acquire low\u2010cost single units from a high-volume manufacturing process, while also creating extra funds to continue support. Success will depend on simple hardware construction to reduce overhead when manufacturing at bulk, and the timely orchestration of a large group of buyers to crowdfund industrial manufacture of the hardware.<sup id=\"rdp-ebb-cite_ref-KohlerFromCollab19_26-0\" class=\"reference\"><a href=\"#cite_note-KohlerFromCollab19-26\">[26]<\/a><\/sup>\n<\/p><p>The initial \u201chardware\u201d phase is the design stage, which commences after the creator group has proven the feasibility of using a particular technology for a desired conservation task. The proven hardware needs to be adapted to an open\u2010source design, and initial investment is required to fund this development. To open source a technology means to apply an open license to it. Various licenses are commonly used to define how OSH design files can be adapted, shared, or commercialized by the user community. (See Table 2.) Each license has its own benefits, depending on the goal. For example, the most open license, CC0, may be useful for community ownership of the design, thus allowing it to be shared, adapted, and commercialized without the need to attribute changes to the original creator(s). Any of the OSH licenses can be used in the framework. Oberloier and Pearce<sup id=\"rdp-ebb-cite_ref-OberloierGeneral17_27-0\" class=\"reference\"><a href=\"#cite_note-OberloierGeneral17-27\">[27]<\/a><\/sup> detail formal procedures for designers of OSH.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 2.<\/b> Comparison of the available Creative Commons OSH licenses\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Creative Commons licenses\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Adaptations can be shared\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Commercial use\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CC0 (No attribution)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Attribution (BY)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ShareAlike (CC BY\u2010SA)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes, but must share alike\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NonCommercial (CC BY\u2010NC)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NonCommercial\u2010NoDerivatives (CC BY\u2010NC\u2010ND)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NoDerivatives (BY\u2010ND)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NonCommercial\u2010ShareAlike (CC BY\u2010NC\u2010SA)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes, but must share alike\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The \u201cmanufacture\u201d phase involves physical fabrication and testing of the subsequent OSH design. Manufacturing often involves complex fabrication processes, including sourcing parts, assembly of printed circuit boards (PCBs), programming the PCBs, and developing housing for the product suitable for field deployment. Keeping this sequence as simple as possible minimizes manufacturing costs. Web services chosen for PCB manufacture should be trialed by testing the build quality of small batches of the developed hardware units before release. Tests should involve checking the quality of assembled PCBs, assessing the fit\u2010for\u2010purpose functionality of the final product, and providing solutions to assembly problems found during manufacture. Issues found at this stage need to be resolved before continuing to the next phase. Spare devices built during this process can be used as trial devices or backup units to support after\u2010sales assurance issues.\n<\/p><p>The \u201crelease\u201d phase consists of the publication, promotion, sales, and large\u2010batch manufacture of the hardware units. Technology design files are first uploaded to a publicly accessible open\u2010source hosting website to allow community access. The guiding principle of this phase is to minimize the cost per unit for users while also generating a pool of funds. Large\u2010scale industrial manufacturing processes reduce unit material cost and assembly time; however, they require an initial sum of money to bulk order before making any sales. To this end, minimum quantity batch manufacturing can be crowdsourced to reduce risk and produce an economy of scale.<sup id=\"rdp-ebb-cite_ref-WheatRaising13_28-0\" class=\"reference\"><a href=\"#cite_note-WheatRaising13-28\">[28]<\/a><\/sup> A crowdfunding platform is required to collect user's funds during funding campaigns. Campaigns should run at regular intervals, and each should end when a maximum order quantity is reached, or a time period is elapsed. The crowdfunding campaign should be publicized on social media and conservation forums, so as to gain traction and reduce the risk of failing to meet the monetary target.\n<\/p><p>The \u201cdistribution\u201d phase consists of the global dissemination of the manufactured units. Depending on the crowdfunding organization used, the logistics group may need to take on the role of global distributor, or employ a dedicated distribution house to ship and track deliveries.\n<\/p><p>The \u201csupport\u201d phase provides technical and logistical assistance to the community of users. Users may require support with diverse issues, including devices getting lost in transit, out\u2010of\u2010box malfunctions, and difficulties with operating the device. A mechanism should be established for users to send their end\u2010of\u2010life devices back to the creator group for re\u2010use or disposal according to the Waste Electrical and Electronic Equipment (WEEE) Regulations.\n<\/p><p>\u201cSustainability\u201d encompasses the reinvestment strategy to facilitate continued functioning of the technology into the future. It involves addressing how pooled funds can be generated and used to continue future development and promote future purchases. To create pooled funds, a margin should be added to each unit sold that generates enough funds from the minimum quantity order to sustain the team until the next group purchase campaign ends. The crowdfunding campaign should set its goal to this minimum batch order. Costs incurred by the logistics group to manage logistical operations, such as employing a part\u2010time member of staff and purchasing packaging for shipping, should be reimbursed from this generated fund. The creator and logistics groups should take a percentage of funds to support their tasks after each group funding campaign. The remaining pool of funds can be held by the logistics group and made accessible to the creator group as needed to pay for further support, bug fixes, and further enhancement or development.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Case_study\">Case study<\/span><\/h2>\n<p>The framework was developed and formalized for the open\u2010source manufacture and distribution of an acoustic monitoring device called AudioMoth, created by <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.openacousticdevices.info\/\" target=\"_blank\">Open Acoustic Devices<\/a> (OAD). Within the management team, the creator group comprised OAD, and the logistics group comprised a small conservation NGO called the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/blog.arribada.org\/\" target=\"_blank\">Arribada Initiative<\/a>.\n<\/p><p>The AudioMoth case study used the Creative Commons Attribution 4.0 International (CC BY) open\u2010source license. (See Table 2.) This attribution allowed full participation by the community in all aspects of design, attributing all changes to the original creators. The license allowed other manufacturers to construct and re\u2010sell the platform independently (e.g., <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labmaker.org\/\" target=\"_blank\">LabMaker<\/a>). This brought mixed benefits: it raised awareness and distribution of devices, while also reducing the funds available for sustaining the devices, although in this case only by about one\u2010tenth.\n<\/p><p>Hardware was designed to fit a single fixed part, consisting of one single\u2010sided PCB that could be purchased from a single manufacturer. The creator group developed the hardware as part of their research budget from U.K. Research Council grants. Development of the product was funded by grants totaling \u223cUSD $13,000 per\/year over three years for two PhD students, excluding stipends and course fees.\n<\/p><p>The chosen manufacturer was <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/circuithub.com\/\" target=\"_blank\">CircuitHub<\/a>. This online web service specializes in assembly of open\u2010source PCBs. It enables design files to be uploaded, shared, manufactured, and delivered to order, with transparent batch size, lead time, and pricing. It first built small test batches of prototype PCBs, which were used to evaluate the single\u2010board construction and address any build issues. Test boards had a 20\u2010day lead time and cost USD $60 per unit for a batch of 50.\n<\/p><p>The finalized AudioMoth design files were published on the open\u2010source hosting websites GitHub and CircuitHub. This made them freely available to adapt and share. CircuitHub's online quotation tool was used to calculate the unit selling price on a manufactured batch of 200 units. A further 20% was added to this unit price, which was calculated to generate enough funds to support global distribution, after\u2010sales support, and future enhancements. Cost\u2010free promotion of AudioMoth was achieved through the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wildlabs.net\/\" target=\"_blank\">WILDLABS<\/a> forum, social media, and the OAD subscription page. Trial devices were distributed to potential users to proliferate the spread of information throughout the conservation community and to generate feedback to the creator group on the AudioMoth utility.\n<\/p><p>The crowdfunding organization <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/groupgets.com\/\" target=\"_blank\">GroupGets<\/a> coordinated the group purchase campaign for each batch order from CircuitHub. GroupGets linked back to OAD's subscription page to help the creators gauge the level of online interest. To fulfill a cost\u2010effective manufacturing run, each campaign had to gain a minimum of 200 backers.\n<\/p><p>GroupGets handled postage to users based in the United States. For orders bound elsewhere, GroupGets batch posted the ordered units to the U.K. headquarters of the logistics group. This minimized individual customs declarations, freeing up sufficient funds from the price margin for the logistics group to distribute units to the rest of the world. The logistics group handled all non\u2010U.S. distribution until the rising level of demand necessitated a dedicated distribution house. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.weengs.co.uk\/\" target=\"_blank\">Weengs<\/a> was then selected to receive devices directly from GroupGets and manage the major distribution channels to the U.K. and E.U.\n<\/p><p>The creator group maintained a website supporting users of AudioMoth. This was updated on a weekly basis, taking approximately two person\u2010hours\/week. The website hosted usage instructions, open\u2010source design files, and a support forum. The forum enabled users to discuss ongoing projects, report technical issues, and suggest modifications to the open\u2010source design files. Over time, it became partially self\u2010sustaining as the community of users gained expertise in the technology. The creator group was still required to provide solutions to more technical issues.\n<\/p><p>The case study supported six rounds of group\u2010purchase campaigns over 18 months, from October 2017 to March 2019. These campaigns generated a pool of funds to finance U.S. distribution, hardware improvements, ongoing user support for all purchased devices, and community\u2010based events, including workshops for users. The sixth group\u2010funding campaign introduced the first hardware modification to the design files. A lesson learned during this change was the importance of maintaining backwards compatible firmware across hardware versions.\n<\/p><p>The six campaigns resulted in 5,242 unit sales in total, with 34% purchased by universities, 23% by conservation organizations, 11% by businesses\/consultants, 1% by government agencies, and 30% by unspecified individuals. All devices sold for USD $49.99 per unit, creating a net revenue of USD $262,048 and a pool of funds after device manufacture and distribution of USD $58,188 (discounting the initial external investment). Pooled funds were managed by the logistics group throughout the study. The remaining pool will continue to be reinvested back into the framework to pay for ongoing maintenance of the website, to support the community of users, and to further improve the hardware. (See Table 3.)\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 3.<\/b> Breakdown of revenue and expenditure throughout the 18\u2010month AudioMoth case study using the framework to support OSH\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Payments in\/out\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Amount (USD)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sale of 5,242 units\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$262,047.58\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Manufacturing costs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2212$141,404.28\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Group purchase fees\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2212$36,871.01\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Postage fees\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2212$11,228.25\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Community activity costs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2212$8,076.19\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Additional hardware improvements\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2212$6,279.67\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total pool of funds\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$58,188.18\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Leveraging_frontline_action\">Leveraging frontline action<\/span><\/h2>\n<p>The case study demonstrates the capability of the framework for sustaining the lifecycle of an open\u2010source conservation tool, while also stimulating global deployment of conservation technology. The framework has to date enabled 687 separate projects to purchase OSH. Purchasers are globally distributed (Figure 2a), with clusters around the financial hubs of advanced economies, particularly in Europe (63%), North America (23%), and Australia (8%). Purchases made elsewhere include Central and South America (4%), Asia (2%), and Africa (<1%).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Hill_ConsLetters2019_12-5.jpg\" class=\"image wiki-link\" data-key=\"6123f4187cf36e56f9f3cf00c562ba4f\"><img alt=\"Fig2 Hill ConsLetters2019 12-5.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/6b\/Fig2_Hill_ConsLetters2019_12-5.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> The global impact of the framework to support OSH: (a) Heat map of all OSH purchase locations; (b) deployment locations and field applications of a 14% sample of purchasers<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Fourteen percent of the framework's end\u2010users posted on social media about conservation work using AudioMoths, with conservation action occurring largely away from purchase locations. For example, Africa was the source country for only two sales, but the deployment location for 23% of purchases. Devices were used for a wide range of applications, including terrestrial monitoring of individual species, general ecosystem soundscape analysis, monitoring of human\u2013wildlife conflicts, experimental marine surveys, and university\u2010level education (Figure 2b). Also by lowering the cost of participating in applied acoustic monitoring research, the application of the framework to AudioMoth facilitated public engagement, including citizen science projects for a national bat survey<sup id=\"rdp-ebb-cite_ref-BCTNational19_29-0\" class=\"reference\"><a href=\"#cite_note-BCTNational19-29\">[29]<\/a><\/sup>, biodiversity mapping surveys<sup id=\"rdp-ebb-cite_ref-PBSoundscapes19_30-0\" class=\"reference\"><a href=\"#cite_note-PBSoundscapes19-30\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BodonPitt19_31-0\" class=\"reference\"><a href=\"#cite_note-BodonPitt19-31\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CetalinguaPartic19_32-0\" class=\"reference\"><a href=\"#cite_note-CetalinguaPartic19-32\">[32]<\/a><\/sup>, and engagement by the general public in wildlife monitoring.<sup id=\"rdp-ebb-cite_ref-GillingsNOCMIG19_33-0\" class=\"reference\"><a href=\"#cite_note-GillingsNOCMIG19-33\">[33]<\/a><\/sup> It also led to the development of new methods to address previously untestable research questions.<sup id=\"rdp-ebb-cite_ref-Pi.C3.B1a.E2.80.90CovarrubiasOpti19_34-0\" class=\"reference\"><a href=\"#cite_note-Pi.C3.B1a.E2.80.90CovarrubiasOpti19-34\">[34]<\/a><\/sup>\n<\/p><p>Those participating in group purchases of AudioMoths actively shared modifications and enhancements, including designs for housing the devices, on independent forum pages as well as on the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.openacousticdevices.info\/support\/\" target=\"_blank\">OAD online community forum<\/a> where changes could be validated by the creator group. Although the framework functions well for AudioMoth, more complicated OSH products may need additional manufacturing stages. Incorporating a framework to support OSH into NGO policy still requires substantial work, including further case studies of different conservation monitoring equipment. Without a framework of the type we propose, however, existing OSH will continue to have short life spans, and remain out of reach for the majority of conservation biologists.\n<\/p><p>Creating legacy value for OSH and proprietary projects used for conservation should be the first principle of action for conservation organizations and universities. Online community platforms for conservation technology such as WILDLABS could take a lead in policy adoption, creating an official platform for recording long\u2010term OSH findings for the community to share and learn from. It is important to push access to OSH toward communities outside of the pockets of wealth and high opportunity that the framework may initially serve. Future developments should investigate benefits and costs of completely transitioning the framework to the conservation community, ways of making it more accessible to those communities typically underserved by citizen science, and methods of community\u2010led support for conservation tools.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We thank Jos\u00e9 Lahoz\u2010Monfort for organizing the ARC CEED workshop entitled \u201cStepping into the future: Driving the Technology Agenda in Ecology and Conservation,\u201d where the OAD and Arribada Initiative seeded the idea for a framework. We thank Stephanie O'Donnell for help with establishing the user\u2010group and providing data for Figure 2b, Anne Dangerfield for logistics management, and Holly Baines for user support. A special thanks goes to the AudioMoth community.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Andrew P. Hill, Peter Prince, Alex Rogers, C. Patrick Doncaster, and Jake L. Snaddon contributed to the design of AudioMoth acoustic sensors, with Alex Rogers, Andrew P. Hill, and Peter Prince leading hardware and software development; Alasdair Davies led the logistics group and AudioMoth distribution service; and all authors contributed to writing of the paper.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work was supported by a U.K. EPSRC Studentship (1658469) to A. Hill, and a U.K. NERC SPITFIRE DTP award (NE\/L002531\/1) to P. Prince. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-RSIllegal18-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RSIllegal18_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">The Royal Society (08 October 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/royalsociety.org\/topics-policy\/projects\/illegal-wildlife-trade\/\" target=\"_blank\">\"Illegal wildlife trade\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/royalsociety.org\/topics-policy\/projects\/illegal-wildlife-trade\/\" target=\"_blank\">https:\/\/royalsociety.org\/topics-policy\/projects\/illegal-wildlife-trade\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Illegal+wildlife+trade&rft.atitle=&rft.aulast=The+Royal+Society&rft.au=The+Royal+Society&rft.date=08+October+2018&rft_id=https%3A%2F%2Froyalsociety.org%2Ftopics-policy%2Fprojects%2Fillegal-wildlife-trade%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UKGovDigital19-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UKGovDigital19_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">U.K. Government (31 January 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.gov.uk\/government\/news\/digital-revolution-to-use-the-power-of-data-to-combat-illegal-wildlife-trade-and-reduce-food-waste\" target=\"_blank\">\"Digital revolution to use the power of data to combat illegal wildlife trade and reduce food waste\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.gov.uk\/government\/news\/digital-revolution-to-use-the-power-of-data-to-combat-illegal-wildlife-trade-and-reduce-food-waste\" target=\"_blank\">https:\/\/www.gov.uk\/government\/news\/digital-revolution-to-use-the-power-of-data-to-combat-illegal-wildlife-trade-and-reduce-food-waste<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Digital+revolution+to+use+the+power+of+data+to+combat+illegal+wildlife+trade+and+reduce+food+waste&rft.atitle=&rft.aulast=U.K.+Government&rft.au=U.K.+Government&rft.date=31+January+2019&rft_id=https%3A%2F%2Fwww.gov.uk%2Fgovernment%2Fnews%2Fdigital-revolution-to-use-the-power-of-data-to-combat-illegal-wildlife-trade-and-reduce-food-waste&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HamptonTheTao15-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HamptonTheTao15_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hampton, S.E.; Anderson, S.S.; Bagby, S.C. et al. (2015). \"The Tao of open science for ecology\". <i>Ecosphere<\/i> <b>6<\/b> (7): 1\u201313. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1890%2FES14-00402.1\" target=\"_blank\">10.1890\/ES14-00402.1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Tao+of+open+science+for+ecology&rft.jtitle=Ecosphere&rft.aulast=Hampton%2C+S.E.%3B+Anderson%2C+S.S.%3B+Bagby%2C+S.C.+et+al.&rft.au=Hampton%2C+S.E.%3B+Anderson%2C+S.S.%3B+Bagby%2C+S.C.+et+al.&rft.date=2015&rft.volume=6&rft.issue=7&rft.pages=1%E2%80%9313&rft_id=info:doi\/10.1890%2FES14-00402.1&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Berger-TalConserv18-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Berger-TalConserv18_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berger-Tal, O.; Lahoz-Monfort, J.J. (2018). \"Conservation technology: The next generation\". <i>Conservation Letters<\/i> <b>11<\/b> (6): e12458. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fconl.12458\" target=\"_blank\">10.1111\/conl.12458<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Conservation+technology%3A+The+next+generation&rft.jtitle=Conservation+Letters&rft.aulast=Berger-Tal%2C+O.%3B+Lahoz-Monfort%2C+J.J.&rft.au=Berger-Tal%2C+O.%3B+Lahoz-Monfort%2C+J.J.&rft.date=2018&rft.volume=11&rft.issue=6&rft.pages=e12458&rft_id=info:doi\/10.1111%2Fconl.12458&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KwokField17-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KwokField17_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kwok, R. (2017). \"Field Instruments: Build it yourself\". <i>Nature<\/i> <b>545<\/b>: 253\u201355. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnj7653-253a\" target=\"_blank\">10.1038\/nj7653-253a<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Field+Instruments%3A+Build+it+yourself&rft.jtitle=Nature&rft.aulast=Kwok%2C+R.&rft.au=Kwok%2C+R.&rft.date=2017&rft.volume=545&rft.pages=253%E2%80%9355&rft_id=info:doi\/10.1038%2Fnj7653-253a&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PearceLab14-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PearceLab14_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pearce, J.M. (2014). \"Laboratory equipment: Cut costs with open-source hardware\". <i>Nature<\/i> <b>505<\/b> (7485): 618. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2F505618d\" target=\"_blank\">10.1038\/505618d<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24476879\" target=\"_blank\">24476879<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+equipment%3A+Cut+costs+with+open-source+hardware&rft.jtitle=Nature&rft.aulast=Pearce%2C+J.M.&rft.au=Pearce%2C+J.M.&rft.date=2014&rft.volume=505&rft.issue=7485&rft.pages=618&rft_id=info:doi\/10.1038%2F505618d&rft_id=info:pmid\/24476879&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AhmadRapid18-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AhmadRapid18_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ahmad, A.; Nadzri, M.M.M.; Rosli, I.M. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/journal.utem.edu.my\/index.php\/jtec\/article\/view\/4354\" target=\"_blank\">\"Rapid Prototyping of Wireless Image Transmission for Wildlife (Tiger) Monitoring System - A Preliminary Study\"<\/a>. <i>Journal of Telecommunication, Electronic and Computer Engineering<\/i> <b>10<\/b> (2-5): 75\u201379<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/journal.utem.edu.my\/index.php\/jtec\/article\/view\/4354\" target=\"_blank\">http:\/\/journal.utem.edu.my\/index.php\/jtec\/article\/view\/4354<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Rapid+Prototyping+of+Wireless+Image+Transmission+for+Wildlife+%28Tiger%29+Monitoring+System+-+A+Preliminary+Study&rft.jtitle=Journal+of+Telecommunication%2C+Electronic+and+Computer+Engineering&rft.aulast=Ahmad%2C+A.%3B+Nadzri%2C+M.M.M.%3B+Rosli%2C+I.M.+et+al.&rft.au=Ahmad%2C+A.%3B+Nadzri%2C+M.M.M.%3B+Rosli%2C+I.M.+et+al.&rft.date=2018&rft.volume=10&rft.issue=2-5&rft.pages=75%E2%80%9379&rft_id=http%3A%2F%2Fjournal.utem.edu.my%2Findex.php%2Fjtec%2Farticle%2Fview%2F4354&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NazirWise17-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NazirWise17_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nazir, S.; Newey, S.; Irvine, R.J. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5226779\" target=\"_blank\">\"WiseEye: Next Generation Expandable and Programmable Camera Trap Platform for Wildlife Research\"<\/a>. <i>PLoS One<\/i> <b>12<\/b> (1): e0169758. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0169758\" target=\"_blank\">10.1371\/journal.pone.0169758<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5226779\/\" target=\"_blank\">PMC5226779<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28076444\" target=\"_blank\">28076444<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5226779\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5226779<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=WiseEye%3A+Next+Generation+Expandable+and+Programmable+Camera+Trap+Platform+for+Wildlife+Research&rft.jtitle=PLoS+One&rft.aulast=Nazir%2C+S.%3B+Newey%2C+S.%3B+Irvine%2C+R.J.+et+al.&rft.au=Nazir%2C+S.%3B+Newey%2C+S.%3B+Irvine%2C+R.J.+et+al.&rft.date=2017&rft.volume=12&rft.issue=1&rft.pages=e0169758&rft_id=info:doi\/10.1371%2Fjournal.pone.0169758&rft_id=info:pmc\/PMC5226779&rft_id=info:pmid\/28076444&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5226779&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SankupellayB4_16-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SankupellayB4_16_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sankupellay, M.; McLaughlin, C.; Davis, J. et al. (2016). \"B4 - Brisbane Backyard Bird Box: Connecting People to the Environment\". <i>Proceedings of the 2016 ACM Conference Companion Publication on Designing Interactive Systems<\/i>: 9\u201312. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2908805.2908808\" target=\"_blank\">10.1145\/2908805.2908808<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=B4+-+Brisbane+Backyard+Bird+Box%3A+Connecting+People+to+the+Environment&rft.jtitle=Proceedings+of+the+2016+ACM+Conference+Companion+Publication+on+Designing+Interactive+Systems&rft.aulast=Sankupellay%2C+M.%3B+McLaughlin%2C+C.%3B+Davis%2C+J.+et+al.&rft.au=Sankupellay%2C+M.%3B+McLaughlin%2C+C.%3B+Davis%2C+J.+et+al.&rft.date=2016&rft.pages=9%E2%80%9312&rft_id=info:doi\/10.1145%2F2908805.2908808&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SoroTheAmbient18-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SoroTheAmbient18_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Soro, A.; Brereton, M.; Dema, T. et al. (2018). \"The Ambient Birdhouse: An IoT Device to Discover Birds and Engage with Nature\". <i>Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems<\/i>: 1\u201313. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3173574.3173971\" target=\"_blank\">10.1145\/3173574.3173971<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Ambient+Birdhouse%3A+An+IoT+Device+to+Discover+Birds+and+Engage+with+Nature&rft.jtitle=Proceedings+of+the+2018+CHI+Conference+on+Human+Factors+in+Computing+Systems&rft.aulast=Soro%2C+A.%3B+Brereton%2C+M.%3B+Dema%2C+T.+et+al.&rft.au=Soro%2C+A.%3B+Brereton%2C+M.%3B+Dema%2C+T.+et+al.&rft.date=2018&rft.pages=1%E2%80%9313&rft_id=info:doi\/10.1145%2F3173574.3173971&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TanEmbed16-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TanEmbed16_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tan, T.F.; Teoh, S.S.; Fow, J.E. et al. (2016). \"Embedded human detection system based on thermal and infrared sensors for anti-poaching application\". <i>Proceedings of the 2016 IEEE Conference on Systems, Process and Control<\/i>: 37\u201342. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FSPC.2016.7920700\" target=\"_blank\">10.1109\/SPC.2016.7920700<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Embedded+human+detection+system+based+on+thermal+and+infrared+sensors+for+anti-poaching+application&rft.jtitle=Proceedings+of+the+2016+IEEE+Conference+on+Systems%2C+Process+and+Control&rft.aulast=Tan%2C+T.F.%3B+Teoh%2C+S.S.%3B+Fow%2C+J.E.+et+al.&rft.au=Tan%2C+T.F.%3B+Teoh%2C+S.S.%3B+Fow%2C+J.E.+et+al.&rft.date=2016&rft.pages=37%E2%80%9342&rft_id=info:doi\/10.1109%2FSPC.2016.7920700&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MainaABio16-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MainaABio16_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maina, C.W.; Muchiri, D.; Njoroge, P. (2016). \"A Bioacoustic Record of a Conservancy in the Mount Kenya Ecosystem\". <i>Biodiversity Data Journal<\/i> <b>4<\/b>: e9906. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2FBDJ.4.e9906\" target=\"_blank\">10.3897\/BDJ.4.e9906<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Bioacoustic+Record+of+a+Conservancy+in+the+Mount+Kenya+Ecosystem&rft.jtitle=Biodiversity+Data+Journal&rft.aulast=Maina%2C+C.W.%3B+Muchiri%2C+D.%3B+Njoroge%2C+P.&rft.au=Maina%2C+C.W.%3B+Muchiri%2C+D.%3B+Njoroge%2C+P.&rft.date=2016&rft.volume=4&rft.pages=e9906&rft_id=info:doi\/10.3897%2FBDJ.4.e9906&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WhytockSolo16-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WhytockSolo16_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Whytock, R.C.; Christie, J. (2017). \"Solo: An open source, customizable and inexpensive audio recorder for bioacoustic research\". <i>Methods in Ecology and Evolution<\/i> <b>8<\/b> (3): 308\u201312. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2F2041-210X.12678\" target=\"_blank\">10.1111\/2041-210X.12678<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Solo%3A+An+open+source%2C+customizable+and+inexpensive+audio+recorder+for+bioacoustic+research&rft.jtitle=Methods+in+Ecology+and+Evolution&rft.aulast=Whytock%2C+R.C.%3B+Christie%2C+J.&rft.au=Whytock%2C+R.C.%3B+Christie%2C+J.&rft.date=2017&rft.volume=8&rft.issue=3&rft.pages=308%E2%80%9312&rft_id=info:doi\/10.1111%2F2041-210X.12678&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KlingDIY18-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KlingDIY18_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kling, J. (2018). \"DIY goes <i>in vivo<\/i>\". <i>Lab Animal<\/i> <b>47<\/b>: 143\u201346. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41684-018-0066-z\" target=\"_blank\">10.1038\/s41684-018-0066-z<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DIY+goes+%27%27in+vivo%27%27&rft.jtitle=Lab+Animal&rft.aulast=Kling%2C+J.&rft.au=Kling%2C+J.&rft.date=2018&rft.volume=47&rft.pages=143%E2%80%9346&rft_id=info:doi\/10.1038%2Fs41684-018-0066-z&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OMahonyTheGov07-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OMahonyTheGov07_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">O'Mahony, S. (2007). \"The governance of open source initiatives: what does it mean to be community managed?\". <i>Journal of Management and Governance<\/i> <b>11<\/b> (2): 139\u201350. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10997-007-9024-7\" target=\"_blank\">10.1007\/s10997-007-9024-7<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+governance+of+open+source+initiatives%3A+what+does+it+mean+to+be+community+managed%3F&rft.jtitle=Journal+of+Management+and+Governance&rft.aulast=O%27Mahony%2C+S.&rft.au=O%27Mahony%2C+S.&rft.date=2007&rft.volume=11&rft.issue=2&rft.pages=139%E2%80%9350&rft_id=info:doi\/10.1007%2Fs10997-007-9024-7&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DrackTheImport18-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DrackTheImport18_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Drack, M.; Hartmann, F.; Bauer, S. et al. (2018). \"The importance of open and frugal labware\". <i>Nature Electronics<\/i> <b>1<\/b> (9): 484\u201386. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41928-018-0133-x\" target=\"_blank\">10.1038\/s41928-018-0133-x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+importance+of+open+and+frugal+labware&rft.jtitle=Nature+Electronics&rft.aulast=Drack%2C+M.%3B+Hartmann%2C+F.%3B+Bauer%2C+S.+et+al.&rft.au=Drack%2C+M.%3B+Hartmann%2C+F.%3B+Bauer%2C+S.+et+al.&rft.date=2018&rft.volume=1&rft.issue=9&rft.pages=484%E2%80%9386&rft_id=info:doi\/10.1038%2Fs41928-018-0133-x&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HsingSustain18-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HsingSustain18_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hsing, P.-Y. (2018). \"Sustainable Innovation for Open Hardware and Open Science \u2013 Lessons from The Hardware Hacker\". <i>Journal of Open Hardware<\/i> <b>2<\/b> (1): 4. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5334%2Fjoh.11\" target=\"_blank\">10.5334\/joh.11<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sustainable+Innovation+for+Open+Hardware+and+Open+Science+%E2%80%93+Lessons+from+The+Hardware+Hacker&rft.jtitle=Journal+of+Open+Hardware&rft.aulast=Hsing%2C+P.-Y.&rft.au=Hsing%2C+P.-Y.&rft.date=2018&rft.volume=2&rft.issue=1&rft.pages=4&rft_id=info:doi\/10.5334%2Fjoh.11&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiUnder18-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiUnder18_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, Z.; Seering, W.; Wallace, D. (2018). \"Understanding Value Propositions and Revenue Models in Open Source Hardware Companies\". <i>Proceedings from the 2018 International Conference on Innovation and Entrepreneurship<\/i>: 214\u201323.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+Value+Propositions+and+Revenue+Models+in+Open+Source+Hardware+Companies&rft.jtitle=Proceedings+from+the+2018+International+Conference+on+Innovation+and+Entrepreneurship&rft.aulast=Li%2C+Z.%3B+Seering%2C+W.%3B+Wallace%2C+D.&rft.au=Li%2C+Z.%3B+Seering%2C+W.%3B+Wallace%2C+D.&rft.date=2018&rft.pages=214%E2%80%9323&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IaconaIdent19-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IaconaIdent19_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Iacona, G.; Ramachandra, A.; McGowan, J. et al. (2019). \"Identifying technology solutions to bring conservation into the innovation era\". <i>Frontiers in Ecology and the Environment<\/i> <b>17<\/b> (10): 591\u201398. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Ffee.2111\" target=\"_blank\">10.1002\/fee.2111<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identifying+technology+solutions+to+bring+conservation+into+the+innovation+era&rft.jtitle=Frontiers+in+Ecology+and+the+Environment&rft.aulast=Iacona%2C+G.%3B+Ramachandra%2C+A.%3B+McGowan%2C+J.+et+al.&rft.au=Iacona%2C+G.%3B+Ramachandra%2C+A.%3B+McGowan%2C+J.+et+al.&rft.date=2019&rft.volume=17&rft.issue=10&rft.pages=591%E2%80%9398&rft_id=info:doi\/10.1002%2Ffee.2111&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PearceEmerging17-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PearceEmerging17_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pearce, J.M. (2017). \"Emerging Business Models for Open Source Hardware\". <i>Journal of Open Hardware<\/i> <b>1<\/b> (1): 2. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5334%2Fjoh.4\" target=\"_blank\">10.5334\/joh.4<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Emerging+Business+Models+for+Open+Source+Hardware&rft.jtitle=Journal+of+Open+Hardware&rft.aulast=Pearce%2C+J.M.&rft.au=Pearce%2C+J.M.&rft.date=2017&rft.volume=1&rft.issue=1&rft.pages=2&rft_id=info:doi\/10.5334%2Fjoh.4&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HillAudio17-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HillAudio17_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hill, A.P.; Prince, P.; Covarrubias, E.P. et al. (2017). \"AudioMoth: Evaluation of a smart open acoustic device for monitoring biodiversity and the environment\". <i>Methods in Ecology and Evolution<\/i> <b>9<\/b> (5): 1199\u20131211. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2F2041-210X.12955\" target=\"_blank\">10.1111\/2041-210X.12955<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=AudioMoth%3A+Evaluation+of+a+smart+open+acoustic+device+for+monitoring+biodiversity+and+the+environment&rft.jtitle=Methods+in+Ecology+and+Evolution&rft.aulast=Hill%2C+A.P.%3B+Prince%2C+P.%3B+Covarrubias%2C+E.P.+et+al.&rft.au=Hill%2C+A.P.%3B+Prince%2C+P.%3B+Covarrubias%2C+E.P.+et+al.&rft.date=2017&rft.volume=9&rft.issue=5&rft.pages=1199%E2%80%931211&rft_id=info:doi\/10.1111%2F2041-210X.12955&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IlinInsights18-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IlinInsights18_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ilin, P.; Platis, D. Hammouda, I. (2018). \"Insights into the Trilateral Relationship of Crowdfunding Campaigns, Open Source and Communities\". <i>Proceedings from the 2018 Open Source Systems: Enterprise Software and Solutions<\/i>: 26\u201336. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-92375-8_3\" target=\"_blank\">10.1007\/978-3-319-92375-8_3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Insights+into+the+Trilateral+Relationship+of+Crowdfunding+Campaigns%2C+Open+Source+and+Communities&rft.jtitle=Proceedings+from+the+2018+Open+Source+Systems%3A+Enterprise+Software+and+Solutions&rft.aulast=Ilin%2C+P.%3B+Platis%2C+D.+Hammouda%2C+I.&rft.au=Ilin%2C+P.%3B+Platis%2C+D.+Hammouda%2C+I.&rft.date=2018&rft.pages=26%E2%80%9336&rft_id=info:doi\/10.1007%2F978-3-319-92375-8_3&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NagleLearning18-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NagleLearning18_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nagle, F. (2018). \"Learning by Contributing: Gaining Competitive Advantage Through Contribution to Crowdsourced Public Goods\". <i>Organization Science<\/i> <b>29<\/b> (4): 547\u2013753. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Forsc.2018.1202\" target=\"_blank\">10.1287\/orsc.2018.1202<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Learning+by+Contributing%3A+Gaining+Competitive+Advantage+Through+Contribution+to+Crowdsourced+Public+Goods&rft.jtitle=Organization+Science&rft.aulast=Nagle%2C+F.&rft.au=Nagle%2C+F.&rft.date=2018&rft.volume=29&rft.issue=4&rft.pages=547%E2%80%93753&rft_id=info:doi\/10.1287%2Forsc.2018.1202&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArlettazFromPub10-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArlettazFromPub10_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Arlettaz, R.; Schaub, M.; Fournier, J. et al. (2010). \"From Publications to Public Actions: When Conservation Biologists Bridge the Gap between Research and Implementation\". <i>BioScience<\/i> <b>60<\/b> (10): 835\u201342. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1525%2Fbio.2010.60.10.10\" target=\"_blank\">10.1525\/bio.2010.60.10.10<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+Publications+to+Public+Actions%3A+When+Conservation+Biologists+Bridge+the+Gap+between+Research+and+Implementation&rft.jtitle=BioScience&rft.aulast=Arlettaz%2C+R.%3B+Schaub%2C+M.%3B+Fournier%2C+J.+et+al.&rft.au=Arlettaz%2C+R.%3B+Schaub%2C+M.%3B+Fournier%2C+J.+et+al.&rft.date=2010&rft.volume=60&rft.issue=10&rft.pages=835%E2%80%9342&rft_id=info:doi\/10.1525%2Fbio.2010.60.10.10&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PocockAVision18-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PocockAVision18_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pocock, M.J.O.; Chandler, M.; Bonney, R. et al. (2018). \"A Vision for Global Biodiversity Monitoring With Citizen Science\". <i>Advances in Ecological Research<\/i> <b>59<\/b>: 169\u2013223. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fbs.aecr.2018.06.003\" target=\"_blank\">10.1016\/bs.aecr.2018.06.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Vision+for+Global+Biodiversity+Monitoring+With+Citizen+Science&rft.jtitle=Advances+in+Ecological+Research&rft.aulast=Pocock%2C+M.J.O.%3B+Chandler%2C+M.%3B+Bonney%2C+R.+et+al.&rft.au=Pocock%2C+M.J.O.%3B+Chandler%2C+M.%3B+Bonney%2C+R.+et+al.&rft.date=2018&rft.volume=59&rft.pages=169%E2%80%93223&rft_id=info:doi\/10.1016%2Fbs.aecr.2018.06.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KohlerFromCollab19-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KohlerFromCollab19_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kohler, T.; Chesbrough, H. (2019). \"From collaborative community to competitive market: The quest to build a crowdsourcing platform for social innovation\". <i>R&D Management<\/i> <b>49<\/b> (3): 356\u201368. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fradm.12372\" target=\"_blank\">10.1111\/radm.12372<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+collaborative+community+to+competitive+market%3A+The+quest+to+build+a+crowdsourcing+platform+for+social+innovation&rft.jtitle=R%26D+Management&rft.aulast=Kohler%2C+T.%3B+Chesbrough%2C+H.&rft.au=Kohler%2C+T.%3B+Chesbrough%2C+H.&rft.date=2019&rft.volume=49&rft.issue=3&rft.pages=356%E2%80%9368&rft_id=info:doi\/10.1111%2Fradm.12372&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OberloierGeneral17-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OberloierGeneral17_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Oberloier, S.; Pearce, J.M. (2017). \"General Design Procedure for Free and Open-Source Hardware for Scientific Equipment\". <i>Designs<\/i> <b>2<\/b> (1): 2. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fdesigns2010002\" target=\"_blank\">10.3390\/designs2010002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=General+Design+Procedure+for+Free+and+Open-Source+Hardware+for+Scientific+Equipment&rft.jtitle=Designs&rft.aulast=Oberloier%2C+S.%3B+Pearce%2C+J.M.&rft.au=Oberloier%2C+S.%3B+Pearce%2C+J.M.&rft.date=2017&rft.volume=2&rft.issue=1&rft.pages=2&rft_id=info:doi\/10.3390%2Fdesigns2010002&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WheatRaising13-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WheatRaising13_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wheat, R.E.; Wang, Y.; Byrnes, J.E.; Ranganathan, J. (2013). \"Raising money for scientific research through crowdfunding\". <i>Trends in Ecology & Evolution<\/i> <b>28<\/b> (2): 71\u20132. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.tree.2012.11.001\" target=\"_blank\">10.1016\/j.tree.2012.11.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Raising+money+for+scientific+research+through+crowdfunding&rft.jtitle=Trends+in+Ecology+%26+Evolution&rft.aulast=Wheat%2C+R.E.%3B+Wang%2C+Y.%3B+Byrnes%2C+J.E.%3B+Ranganathan%2C+J.&rft.au=Wheat%2C+R.E.%3B+Wang%2C+Y.%3B+Byrnes%2C+J.E.%3B+Ranganathan%2C+J.&rft.date=2013&rft.volume=28&rft.issue=2&rft.pages=71%E2%80%932&rft_id=info:doi\/10.1016%2Fj.tree.2012.11.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BCTNational19-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BCTNational19_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bat Conservation Trust (2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bats.org.uk\/our-work\/national-bat-monitoring-programme\" target=\"_blank\">\"National Bat Monitoring Programme\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bats.org.uk\/our-work\/national-bat-monitoring-programme\" target=\"_blank\">https:\/\/www.bats.org.uk\/our-work\/national-bat-monitoring-programme<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=National+Bat+Monitoring+Programme&rft.atitle=&rft.aulast=Bat+Conservation+Trust&rft.au=Bat+Conservation+Trust&rft.date=2019&rft_id=https%3A%2F%2Fwww.bats.org.uk%2Four-work%2Fnational-bat-monitoring-programme&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PBSoundscapes19-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PBSoundscapes19_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Point Blue Conservation Science (2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.soundscapes2landscapes.org\" target=\"_blank\">\"Soundscapes to Landscapes\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.soundscapes2landscapes.org\" target=\"_blank\">https:\/\/www.soundscapes2landscapes.org<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Soundscapes+to+Landscapes&rft.atitle=&rft.aulast=Point+Blue+Conservation+Science&rft.au=Point+Blue+Conservation+Science&rft.date=2019&rft_id=https%3A%2F%2Fwww.soundscapes2landscapes.org&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BodonPitt19-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BodonPitt19_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bodon, S. (26 February 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wesa.fm\/post\/pitt-researchers-are-eavesdropping-birds-name-science\" target=\"_blank\">\"Pitt Researchers Are Eavesdropping On Birds In The Name Of Science\"<\/a>. <i>90.5 WESA<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.wesa.fm\/post\/pitt-researchers-are-eavesdropping-birds-name-science\" target=\"_blank\">https:\/\/www.wesa.fm\/post\/pitt-researchers-are-eavesdropping-birds-name-science<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Pitt+Researchers+Are+Eavesdropping+On+Birds+In+The+Name+Of+Science&rft.atitle=90.5+WESA&rft.aulast=Bodon%2C+S.&rft.au=Bodon%2C+S.&rft.date=26+February+2019&rft_id=https%3A%2F%2Fwww.wesa.fm%2Fpost%2Fpitt-researchers-are-eavesdropping-birds-name-science&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CetalinguaPartic19-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CetalinguaPartic19_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Cetalingua (2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cetalingua.com\/citizen-science\/\" target=\"_blank\">\"Participatory Platform for Citizen Scientists\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cetalingua.com\/citizen-science\/\" target=\"_blank\">https:\/\/www.cetalingua.com\/citizen-science\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Participatory+Platform+for+Citizen+Scientists&rft.atitle=&rft.aulast=Cetalingua&rft.au=Cetalingua&rft.date=2019&rft_id=https%3A%2F%2Fwww.cetalingua.com%2Fcitizen-science%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GillingsNOCMIG19-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GillingsNOCMIG19_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Gillings, S.; Moran, N. (2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nocmig.com\/audiomoth\" target=\"_blank\">\"AudioMoth\"<\/a>. <i>NOCMIG: Tips for Recording Nocturnal Bird Migration<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nocmig.com\/audiomoth\" target=\"_blank\">https:\/\/www.nocmig.com\/audiomoth<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=AudioMoth&rft.atitle=NOCMIG%3A+Tips+for+Recording+Nocturnal+Bird+Migration&rft.aulast=Gillings%2C+S.%3B+Moran%2C+N.&rft.au=Gillings%2C+S.%3B+Moran%2C+N.&rft.date=2019&rft_id=https%3A%2F%2Fwww.nocmig.com%2Faudiomoth&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Pi.C3.B1a.E2.80.90CovarrubiasOpti19-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Pi.C3.B1a.E2.80.90CovarrubiasOpti19_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pi\u00f1a\u2010Covarrubias, E.; Hill, A.P.; Prince, P. et al. (2019). \"Optimization of sensor deployment for acoustic detection and localization in terrestrial environments\". <i>Remote Sensing in Ecology and Conservation<\/i> <b>5<\/b> (2): 180\u201392. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Frse2.97\" target=\"_blank\">10.1002\/rse2.97<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Optimization+of+sensor+deployment+for+acoustic+detection+and+localization+in+terrestrial+environments&rft.jtitle=Remote+Sensing+in+Ecology+and+Conservation&rft.aulast=Pi%C3%B1a%E2%80%90Covarrubias%2C+E.%3B+Hill%2C+A.P.%3B+Prince%2C+P.+et+al.&rft.au=Pi%C3%B1a%E2%80%90Covarrubias%2C+E.%3B+Hill%2C+A.P.%3B+Prince%2C+P.+et+al.&rft.date=2019&rft.volume=5&rft.issue=2&rft.pages=180%E2%80%9392&rft_id=info:doi\/10.1002%2Frse2.97&rfr_id=info:sid\/en.wikipedia.org:Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance. The Iacona <i>et al.<\/i> in-press article was updated with the now-published version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204405\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.577 seconds\nReal time usage: 1.593 seconds\nPreprocessor visited node count: 25074\/1000000\nPreprocessor generated node count: 34050\/1000000\nPost\u2010expand include size: 161473\/2097152 bytes\nTemplate argument size: 54341\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 510.315 1 - -total\n 87.21% 445.066 1 - Template:Reflist\n 74.96% 382.541 34 - Template:Citation\/core\n 64.05% 326.875 27 - Template:Cite_journal\n 14.60% 74.514 7 - Template:Cite_web\n 7.47% 38.097 1 - Template:Infobox_journal_article\n 7.11% 36.282 1 - Template:Infobox\n 5.55% 28.311 28 - Template:Citation\/identifier\n 4.82% 24.573 80 - Template:Infobox\/row\n 3.82% 19.505 36 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11407-0!*!0!!en!5!* and timestamp 20200707204404 and revision id 37782\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware\">https:\/\/www.limswiki.org\/index.php\/Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","d14ab23a070a6dedd70edd1f31559d55_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ea\/Fig1_Hill_ConsLetters2019_12-5.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/6b\/Fig2_Hill_ConsLetters2019_12-5.jpg"],"d14ab23a070a6dedd70edd1f31559d55_timestamp":1594154644,"3ab9770e7fca9810f5573456215e3853_type":"article","3ab9770e7fca9810f5573456215e3853_title":"What is the \"source\" of open-source hardware? (Bonvoisin et al. 2017)","3ab9770e7fca9810f5573456215e3853_url":"https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_%22source%22_of_open-source_hardware%3F","3ab9770e7fca9810f5573456215e3853_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:What is the \"source\" of open-source hardware?\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nWhat is the \"source\" of open-source hardware?Journal\n \nJournal of Open HardwareAuthor(s)\n \nBonvoisin, J\u00e9r\u00e9my; Mies, Robert; Boujut, Jean-Fran\u00e7ois; Stark, RainerAuthor affiliation(s)\n \nTechnische Universit\u00e4t Berlin, Grenoble Alpes UniversityPrimary contact\n \nEmail: jeremy dot bonvoisin at tu-berlin dot deYear published\n \n2017Volume and issue\n \n1(1)Page(s)\n \n5DOI\n \n10.5334\/joh.7ISSN\n \n2514-1708Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/openhardware.metajnl.com\/articles\/10.5334\/joh.7\/Download\n \nhttps:\/\/openhardware.metajnl.com\/articles\/10.5334\/joh.7\/galley\/12\/download\/ (PDF)\n\n\n\n\n \n This article contains rendered mathematical formulae. You may require the TeX All the Things plugin for Chrome or the Native MathML add-on and fonts for Firefox if they don't render properly for you. \n\n\nContents\n\n1 Abstract \n2 Introduction \n3 The context of open-source hardware \n4 Definitions of openness from practice and research \n\n4.1 Definitions from practice communities \n4.2 Definitions from scholarly research \n4.3 Theory-practice-gap \n\n\n5 Research approach and methodology \n\n5.1 Focus: Complex non-electronic hardware products \n5.2 Product selection \n5.3 Data acquisition \n5.4 Openness assessment criteria \n5.5 Clustering \n\n\n6 Results \n\n6.1 Pool of gathered products \n6.2 Overall openness of the published documentation \n6.3 Relative frequency of openness factors \n6.4 Role of contextual criteria in openness \n6.5 Typical profiles \n\n\n7 Discussion \n\n7.1 Framing the \u201csource\u201d of open source hardware \n7.2 Limits of the reported research \n\n\n8 Conclusion \n9 Footnotes \n10 Acknowledgements \n\n10.1 Competing interests \n\n\n11 References \n12 Notes \n\n\n\nAbstract \nWhat \u201copen source\u201d means once applied to tangible products has been so far mostly addressed through the light of licensing. While this approach is suitable for software, it appears to be over-simplistic for complex hardware products. Whether such a product can be labelled as open-source is not only a question of licence but a question of documentation, i.e. what is the information that sufficiently describes it? Or in other words, what is the \u201csource\u201d of open-source hardware? To date there is no simple answer to this question, leaving large room for interpretation in the usage of the term. Based on analysis of public documentation of 132 products, this paper provides an overview of how practitioners tend to interpret the concept of open-source hardware. It specifically focuses on the recent evolution of the open-source movement outside the domain of electronics and DIY to that of non-electronic and complex open-source hardware products. The empirical results strongly indicate the existence of two main usages of open-source principles in the context of tangible products: publication of product-related documentation as a means to support community-based product development and to disseminate privately developed innovations. It also underlines the high variety of interpretations and even misuses of the concept of open-source hardware. This reveals in turn that this concept may not even be clear to practitioners and calls for more narrowed down definitions of what has to be shared for a product to be called open source. This article contributes towards this effort through the definition of an open-source hardware lifecycle, summarizing the observed approaches to open-source hardware.\nKeywords: open-source hardware, open design, open innovation, open-source innovation, open-source product development\n\nIntroduction \nWe are currently witnessing an increasing number of initiatives transferring product development and production from the private sector to the public. Enabled by the growing accessibility of affordable manufacturing technology, this is manifested in the expansion of the so-called \u201cmaker culture,\u201d which takes action to install participational production as an alternative to industrial production.[1][2] The emergence of this culture is interwoven with the phenomenon of open-source hardware (OSH), which transfers open-source principles (as defined by Open Source Initiative 2007) from their origins in software development to the world of physical objects.[3] While these new practices are raising significant attention, they are still in their infancy and struggle to reveal their full economic, social, and environmental potential. One of the challenges they face is that sharing knowledge about atoms is not as frictionless as sharing bits.\nBoth practitioners and the scientific community generally acknowledge that online sharing of a piece of hardware is more difficult than the sharing of a piece of software (e.g., see discussion of this point by Raasch[4]. Software is digital by nature; it is made of series of characters in a format that can be shared and displayed online without specific tools, with a text editor being enough. Hardware may need to be described through more complex constructs like 2D or 3D schematics, which may require more specific software to be edited and displayed. Based on the evaluation of a pool of 20 OSH projects whose products embedded both software and hardware components, Balka, Raasch, and Herstatt[5] highlighted that hardware components were generally less documented than the software components. This result raises questions in terms of practice. When a piece of hardware is poorly documented, is it still open source? What does \u201cless documented\u201d mean? What are the minimal requirements for labeling a hardware product as \"open source\"?\nIn the absence of clear guidance on this issue, it is not easy to draw a line between which piece of hardware is open source and which is not, even when licensing terms may be clear. Unlike in software, attributing appropriate licences[a] is not sufficient to call hardware open source. Given OSH is a sociotechnical phenomenon, the answer primarily depends on how the product documentation enables co-development and replication. This article seeks to provide guidance on which information sufficiently describes OSH. In other words, what is the source of OSH?\nThe objective of this article is to provide an overview of how current projects tend to interpret and make use of the concept of OSH. Its ultimate goal is to provide a deeper description of what OSH means based on the observation of actual practices. This is performed through the analysis of the \u201csource\u201d, i.e., the published documentation of 132 OSH products with the help of categorical criteria addressing the question \u201chow open are OSH products?\u201d It specifically focuses on the recent evolution of the open-source movement outside the domain of electronics and DIY to those of non-electronic and complex OSH products.\nThe remainder of the paper is structured as follows. In the next section the general context of emergence of OSH as an alternative product development pattern based on free distribution of information is depicted and characterized. Then definitions from practice communities and scholars are analysed and combined in order to provide a consolidated overview of the concept of OSH. After the definitions, the methodological approach for the acquisition of empirical data allowing the analysis of current practices of OSH documentation is introduced. The results produced by the application of this method are then described and interpreted. Finally, we summarize the findings into an original framework termed \"OSH lifecycle,\" additionally summarizing observed approaches to OSH.\n\nThe context of open-source hardware \nOSH is a relatively young phenomenon with projects emerging in the past decade[7], although it has several prominent examples already. Pioneering projects such as RepRap, Open Source Ecology, and Local Motors have certainly set a precedence to lift the air of mystery and aloofness of engineering ingenuity closely guarded for means of commercial appropriation. As to whether these are heralds of what Moritz et al.[8] depict as disruptive changes on the upstream end of value chains toward value-co-creation, only time will tell. Clearly, this alternative course of action could take an active part in shaping the technological future. Indeed, there are already promising examples of successful businesses based on OSH for which the scientific community identified corresponding value creation models.[9][10] These examples, together with the empirical evidence provided by free and open-source software, allow for a foreseeing of a flourishing future for OSH.[9][10]\nLike free and open-source software, OSH is an IT-enabled internet phenomenon. Fjeldsted et al.[11], as well as Bonvoisin and Boujut[12], point out the integral part IT platforms play in fostering product-related data sharing and community-based product development, as well as the emergence of OSH-based business models. However, Raasch[4] draws a contrast: compared with OSS development, the aspect of physical object design in OSH has strong impacts on required skills, tools, and infrastructure. This aspect is even increasingly salient as the focus of OSH progressively expands towards many other forms of hardware than electronic hardware, like mechanical, construction, medical, optical, agricultural, or textile hardware. From a design point-of-view, the degree of freedom in the problem-solution space introduced by their mechanical portion rises significantly. Howard et al.[13] and Raasch, Herstatt, and Balka[14] also illustrate how OSH is looking at the full spectrum of product complexity and different manufacturing strategies, from do-it-yourself (DIY) to industrial production.\nThe capacity of communities outside the closed and hierarchical environment of research and development (R&D) departments to develop complex and high-quality products remains a challenge; the largest majority of OSH products available to date are gadgets for hobbyists.[15] As processes transform inputs into outputs, their structure is generally related to the product and project scope. OSH can be described as individual or participatory realization of a shared product design. Therefore, 3D printing designs shared by individuals on sites such as Thingiverse[b] may classify as OSH, similar to large community-based efforts such as the E-Nable project.[c] This makes a big difference in regards to the type of effort being structured for the realization of an open-source product. Hence, depending on the product scope and also the originator's intent to foster co-design, levels of interaction maturity differ starkly in OSH. A useful macro-level classification is offered by Camarinha-Matos et al.[16], starting at networks, then moving to coordination, cooperation, and collaboration.\nThe conceptual space that supports collective spirit and collaborative problem solving is described by Maher, Paulini, and Murty.[17] It is based on the three aspects of shared representation, motivation, and communication. They enable problem solving in collective design projects within a continuum from collected (individual) to collective intelligence. The latter is demarcated by collaborative generation of solutions as well as synthesis of individual solutions. From their study of collective design activities[18] conclude that within the frame of what they call an inclusive nature of participatory design, individuals proactively self-organize and choose their own roles as well as period of involvement. The maturity of this participatory nature may in fact be the primary determinant of collective design projects.\nSocial dynamics in crowds are context-dependent, however. In the OSH context, from all of the above, communities can be viewed as socially formed groups of heterogeneous actors who co-create OSH products. Depending on the participatory roles of actors, they may engage as followers, replicators, developers, or community managers. In the frame of open innovation, West and Lakhani[19] describe them as actor volunteers lacking of organizational affiliation. Due to partial influence of market-based factors, this role may somewhat reflect the notion of people giving away their time and effort without any monetary compensation. In addition, actors may as well include individuals, firms, and many other types of organizations.[20]\nAksulu and Wade[21] provide insight on how pure open-source systems set a community-based context of personal development, process learning, and effective technology outputs. In contrast, within the purely proprietary context, purpose is realized by the efficient generation of technology outputs within organizations with clear boundaries.\nWhat exactly makes open-source processes effective seems to be rooted in the task environment. According to Lee and Cole[22], the trade-off between exploration and exploitation is made through a two-tier task structure of generated variations within the periphery, and selection and retention at the core. In their case study of the Linux kernel development project, they observe community-based knowledge creation in the form of an evolutionary learning process based on a culture of critical evaluations and peer learning. Moreover, Howison and Crowston[23] describe collaborative development of OSS as an evolutionary process of incremental and independent tasks, by which a layer structure emerges task-by-task. They argue that added layers change the context over time and significantly reduce complexity of tasks. By gradually laying the foundation, previously unattainable tasks eventually may suddenly become easily achievable by individual developers, or simple workarounds may become obvious. This is in line with the concept of Maher et al.[24] concerning the co-evolution of the problem and solution spaces, which describes how these dimensions influence each other during the design process. Collaborative evolutionary design on the activity level is key in explaining how solutions are generated effectively in open-source systems[d] and needs to be further researched.\n\nDefinitions of openness from practice and research \nThis section reviews existing definitions from practice and identifies resulting implications in terms of published product-related documentation. These definitions are then matched with existing contributions from research in order to provide a comprehensive framework defining OSH. Results of empirical studies which challenge these definitions and show a more heterogeneous landscape of OSH are then presented and lead to the identification of two research questions addressed in this article.\n\nDefinitions from practice communities \nThe Open Source Hardware Statement of Principles 1.0 states that \u201copen source hardware is hardware whose design is made publicly available so that anyone can study, modify, distribute, make, and sell the design or hardware based on that design.\u201d[25] It is based on the assumption that publishing a \u201cdesign\u201d (alternatively termed \u201cdocumentation\u201d) realizes the four freedoms of the open-source concept, which are reinterpreted in the context of tangible products as the freedom to study, modify, make, and distribute.[e]\nThis definition is an adaptation of the original Open Source Definition by the Open Source Initiative[27] to the realms of tangible products. Coined in the context of free and open-source software, it specifies requirements on two interwoven objects: the \u201csoftware,\u201d alternatively called the \u201cproduct,\u201d and the \u201csource code.\u201d It is implicitly accepted that the object called \u201csource code\u201d allows defining the object called \u201cproduct\u201d unambiguously in its depth and its entirety. In the case of software, this implicit prerequisite is automatically fulfilled, as a software product is the translation of a text written in programming language into a machine language by a deterministic algorithm. However, this is not the case for tangible products: what information has to be shared in order to allow any interested person to study, modify, make, and distribute a piece of hardware is not a simple question.\nA closer look at the definitions of widely recognized OSH licences and current practices of OSH seeks to give some hints and shows that the four freedoms of open source tend to be supported by different document types or properties[28]:\n\n The freedom to study (i.e., the right to access sufficient information to understand how the piece of hardware\u2014referred herein as the product\u2014works and to retrace the underlying design rationale as defined by Wang, Johnson, and Bracewell[29]) can be supported by the publication of schematics, 2D, or 3D CAD files.\n The freedom to modify (i.e., the right to edit the product definition documents and to tweak or develop the product further for any purpose) can be supported by the publication of all documents in their original editable format.\n The freedom to make (i.e., the right to use the product definition documents in order to make\u2014in other words to produce, to manufacture\u2014the piece of hardware) can be supported by the publication of a bill of materials and assembly instructions.\n The freedom to distribute (i.e., the right to give or sell the product definition documents as well as the physical products fabricated with the help of these documents) is allowed by the publication of all documents under a licence which grants free redistribution, including for commercial purposes.\nDefinitions from scholarly research \nScholarly definitions\u2014mostly stemming from innovation management research\u2014deliver categories which are consistent with the Open Source Hardware Statement of Principles 1.0 without, however, giving further details on the nature of the documentation to be published.\nFreedoms to study, to edit, and to make are consistent with the three factors of openness defined by Balka, Raasch, and Herstatt.[5] They respectively term these factors \"transparency,\" \"accessibility,\" and \"replicability.\" Transparency refers to the possibility for any interested person to access sufficient information to understand the product in detail without restriction (which equals the freedom to study). Accessibility refers to the possibility for any interested person to edit design information and therefore to further develop the product (which equals the freedom to modify). Replicability refers to the possibility for any interested person to physically produce the product (which equals the freedom to make). These three factors of openness are introduced as preconditions for three complementary behaviors on the part of the community surrounding the product: Transparency enables observation (and eventually feedback), accessibility enables further development (and eventually co-development), and replicability enables prototyping and production (and eventually co-production). What is missing in this contribution is the fourth factor of openness termed henceforth \"commercial usability,\" which addresses the above listed freedom to distribute.\nThe four freedoms and corresponding factors of openness cover both \u201cdistinct and competing meanings\u201d of openness identified by von Hippel[30]: the permeability of the innovation process to the participation of external people and the public sharing of documentation. Transparency, replicability, and commercial usability are about sharing documentation publicly. Accessibility is about enabling the participation of external people in the design process. Aitamurto, Holland, and Hussain[31] term two meanings of openness: process openness (whether the innovation process is open or closed) and product openness (whether the innovation outcome is open or closed). Huizingh[32] defines open-source innovation as a result of both process and product openness. Raasch, Herstatt, and Balka[14] particularly deliver the following definition: \u201copen-source innovation is characterized by free revealing of information on a new design with the intention of collaborative development of a single design or a limited number of related designs for market or nonmarket exploitation.\u201d\nThe coherent picture of OSH drawn by those definitions from practice and scholarly research is displayed in Figure 1. OSH builds upon product and process openness. Product openness results from the factors transparency, replicability, and commercial usability, which respectively address the freedoms to study, make, and distribute. Process openness results from the sole factor accessibility, which addresses the freedom to edit.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Forms of openness involved in open-source hardware\n\n\n\nTheory-practice-gap \nThe few existing empirical studies of OSH deliver a picture from practice observations that is less strict than the ambitious theoretical approach summarized by Figure 1, beyond the misuses, which can be expected with the upcoming of a new concept.\nBalka, Raasch, and Herstatt[5] showed that factors of openness may be valued differently by development community members in the fields of consumer electronics and IT hardware. In the communities they analyzed, accessibility of software tended to play the largest role in the involvement of members, while transparency and replicability of software played a secondary role, and the openness of the related hardware played no role. They also highlighted that, as a result of the low importance given to hardware in those communities, hardware components were generally \u201cless documented\u201d than the software components. This raises the following research question:\n\nRQ1 \u2013 To what extent are the four aspects of openness reflected by practitioners in the publication of product-related documentation? Do practitioners tend to employ the whole set of these factors, or do they tend to employ only subsets of these factors?\nBased on interviews with originators of OSH products, Bonvoisin et al.[33] defined the contours of two archetypes of projects developing OSH they termed \"development communities\" and \"isolated innovators.\" Development communities are characterized by high process openness, adopt systematic and early release policies, and gather contributions from a larger and permeable group of people. On the contrary, isolated innovators are characterized by low levels of process openness. They do not gather contributions from the outside, but rather they release complete product documentations after product versions are fully developed. While the former are interested in integrating people from the outside during the product development process, the latter are more interested in broadcasting their innovation, i.e., enabling other people to produce it. This contradicts the statement made by Gacek and Arief[34] that \u201cthe term 'open source' has been widely used to describe a software development process.\u201d Findings reported by Bonvoisin et al.[33] suggest on the contrary that motivations for publishing documentation may depend on the phase of the product lifecycle in which the community is supposed to play a role and raise the following second research question:\n\nRQ2 \u2013 If factors of openness are understood by practitioners as optional, can we find typical patterns in the way practitioners decide to comply with specific factors of openness? Are these aspects related to specific product lifecycle phases?\nResearch approach and methodology \nIn order to address the research questions defined above, an empirical approach based on the systematic analysis of a representative group of OSH products, as well as their published documentation, has been adopted. This approach is similar to those adopted by Balka, Raasch, and Herstatt[5][35] while avoiding two pitfalls limiting the validity of the produced results. The first is the use of unclear categories such as the size of the development community. As shown in Bonvoisin et al.[33], belonging to a development community is a fickle concept which may be based on different types of activities (from pressing the button \u201cI like\u201d to bringing significant contribution to the product design), which are highly flexible over time and therefore difficult to measure objectively.[f] The second is the use of subjective evaluation of the openness factors with the help of Likert scales. In contrast to the approaches of Balka, Raasch, and Herstatt[5][35], the research reported here strives at a systematic analysis of published documentation with the help of measurable indicators, from the part of an impartial observer.\nIn the following subsections, the approach adopted for the acquisition and the analysis of empirical data is presented. The first subsection introduces methodological choice made in focusing on non-electronic complex hardware products. The second subsection describes the method applied for the establishment of a representative pool of those products. The third subsection defines a set of criteria used to characterize products and their related documentation. The fourth subsection defines criteria for the assessment of openness factors based on the acquired data. The last subsection introduces a clustering method used for identifying typical patterns in the publication of OSH documentation.\n\nFocus: Complex non-electronic hardware products \nWhile the transfer of open-source principles from software to hardware historically and logically started with electronic hardware[36], other technologies are increasingly impacted by the phenomenon. The extension of open-source practices to non-electronic hardware such as mechanical products is of particular interest in terms of documentation. Electronic hardware is a thoroughly standardized field where components are purchased off the shelf. But this is not the case for mechanical hardware, where non-standardized free form components play a large role.\nAlso, particularly interesting is the consideration of complex products. Product complexity is defined by Jacobs[37] as \u201ca design state resulting from the multiplicity of, and relatedness among, product architectural elements.\u201d This characteristic influences the level of professionalization required in the development and production of a given product. It also determines whether production can take place in either DIY or industrial production settings. Noteworthy is that, while DIY and OSH are two interwoven phenomena, not every OSH product is meant to be produced in a DIY production setting.[38] Product complexity also relates to design effort in terms of resources consumed and process duration.[39] Highly complex products tend to require inputs from multiple people and are more relevant for the topic of collaborative design.\nIn order to reflect the challenges faced by OSH products in terms of documentation, the methodological approach pursued in this article is to focus on the most challenging range of products. The underlying assumption is that \u201cwho can do more can do less,\u201d so what applies to complex non-electronic hardware applies also to more electronic hardware and simple products. As a consequence, this article specifically focuses on the recent evolution of the open-source movement outside the domain of electronics and DIY to those of non-electronic and complex OSH products. It excludes the field of purely electronic hardware products. Not only is this methodological choice in consistency with the academic background of the authors situated in mechanical engineering and collaborative engineering design; it is also in line with the most recent advancement of the open-source concept on forms of hardware which are in the sphere of influence in these disciplines. It should be borne in mind that the huge achievements of the still much larger open electronic hardware field have paved the way for this development. In no way is the exclusion of this field in this study an indicator of its irrelevance. Rather, it is a matter of limiting the extent of the research undertaking.\nAs a result of this position, in the rest of the paper, \u201copen source hardware\u201d is used to refer to complex, non-electronic OSH.\n\nProduct selection \nIn contrast to previous research works aiming at grasping the contours of a phenomenon related to open source, e.g., sharing of 3D-printed designs (see Kyriakou, Nickerson, and Sabnis[40]), no major centralized online entry point exists which aggregates relevant products in the context of OSH. While platforms such as GitHub or Thingiverse are playing this role for free and open-source software or 3D-printing communities, to date, there is no generally acknowledged and widely used online platform supporting open-source product development. These observations have been respectively made by Raasch[4] and Howard et al.[13] and are still valid. On the contrary, projects developing OSH products tend to use a wide variety of tools for collaboration and dissemination.[33]\nAs a result, OSH products have been identified using conventional internet search engines using a snowball-effect approach. The results of this search have been screened through a restrictive criteria set in order to ensure a conservative evaluation of the phenomenon to be studied and to keep a deliberately narrow focus within the highly diverse field of open-source innovation (see for example Ehls[41] on that topic). The following selection criteria have been applied:\n\n The product is the outcome of discrete manufacturing. Products of food and process industries such as yogurt, cement, chemicals, or plastic compounds are excluded.\n The product contains at least in-house, tangible, and non-electronic hardware, which includes mechanical or any other type of non-electronic physical elements (e.g., textile). It may eventually include electronic hardware and consequently software. Purely electronic hardware or software products such as Arduino or Linux are therefore excluded. In-house means that the piece of non-electronic hardware is \"original equipment\" created by the product originator and not a component bought off the shelf.\n The product is complex. While well-defined metrics assessing product complexity on a positive real scale have been proposed in the scientific literature (e.g., Rodriguez-Toro, Jared, and Swift [39]), using those metrics requires having sufficient access to detailed data such as the number of parts. This condition is difficult to satisfy in the context of observation of partly documented and defined products. Therefore, in the following analysis, the complexity of products is evaluated with a rule of thumb: products are considered as complex if they consist of more than two parts of different materials. Products such as business card holders, cell phone cases, or other 3D-printed gimmicks are out of scope. The objective here is to bring to the foreground those open-source products that are on the upper side of the complexity scale. In other words, to select the few complex ones and let aside the myriad of gimmicks that can be found on CAD file exchange platforms such as Thingiverse or Shapeways.\n The product is developed for functional rather than aesthetic purposes. Jewelry, artistic, and decorative items do not fulfill this criterion and therefore were not included.\n The product is at least partly defined, i.e., is provided with sufficient documentation to indicate that the product development maturity is equivalent or above the stage \u201csystem-level design\u201d of the product development process as defined by Ulrich and Eppinger.[42] Undeveloped product concepts are not considered. Thus, the focus delineates them from challenge platforms, which focus solely on capturing concepts from external participants.\n The product is labelled by its surrounding community as open source. The terms \u201copen source,\u201d \u201copen-source hardware,\u201d or \u201copen hardware\u201d are used in the published documentation as an adjective to qualify either the product or the activities around the product. This is quite a conservative criterion. A non-negligible number of projects adopt principles of OSH without necessarily fitting neatly into this conservative criterion.\n There are no systematic 1:1 relations between product, project, and community. For example, a community may be involved in more than one project and a project may involve the development of more than one product. However, in order to simplify the analysis, only one product per project and per community has been considered.\nData acquisition \nFor each of the selected products, two sets of descriptive criteria are defined. These are summarized in Table 1. The first set is dedicated to the characterization of the published product-related information. It translates in measurable terms the best practices of OSH as provided by the Open Source Hardware Association[43] and complemented by a practice review performed prior to the reported research.[28] In the absence of practicable indicators for assessing the quantity, the quality, or the relevance of a piece of information, simple binary criteria are used, which indicate the presence or the absence of a given piece of information or of a given property. The second set delivers contextual information about the product, the corresponding development project, as well as the surrounding product development community.\n\n\n\n\n\n\n\nTable 1. Open-source hardware product characterization criteria.\n\n\nRef.\n\nName\n\nDescription\n\n\nPart I \u2013 Criteria regarding shared product-related documentation\n\n\na\n\nCAD files available\n\nBoolean value indicating whether CAD files or schematics of the non-electronic hardware are available online\n\n\nb\n\nCAD files aeditable\n\nBoolean value indicating whether the online released CAD files of the product are editable. CAD files are considered editable if they are released in their original format. They are not considered editable if they are only released in an export format such as PDF or STL, which does not allow further modifications.\n\n\nc\n\nAssembly instructions available\n\nBoolean value indicating whether instructions for building the non-electronic hardware are available online\n\n\nd\n\nAssembly instructions editable\n\nBoolean value indicating whether the published assembly instructions are editable. Assembly instructions are considered editable if they can be edited in a web 2.0 environment or downloaded as editable files. A file is furthermore considered editable if it is released in its original format. It is not considered as editable if it is only available in an export format such as PDF.\n\n\ne\n\nBill of materials available\n\nBoolean value indicating whether a bill of materials relative to the non-electronic hardware is available online\n\n\nf\n\nBill of materials editable\n\nBoolean value indicating whether the published bill of materials is editable. A bill of materials is considered editable if it can be edited in a web. 2.0 environment or downloaded as an editable file. A file is furthermore considered editable if it is released in the original format. It is not considered editable if it is only available in an export format such as PDF.\n\n\ng\n\nGuidelines for participation\n\nBoolean value indicating whether guidelines for participation or a dedicated call for contribution are provided to potential contributors\n\n\nh\n\nCommercial usage allowed\n\nBoolean value indicating whether the licence applied to the non-electronic hardware allows commercial usage of the published content. If no licence is applied, the criterion is set to false.\n\n\nPart II \u2013 Criteria regarding contextual information\n\n\ni\n\nLicense\n\nLicensing scheme used for the publication of the non-electronic hardware\n\n\nj\n\nContains electronics\n\nBoolean value indicating whether the product contains electronic hardware components as well\n\n\nk\n\nMaturity\n\nDefines the maximum maturity level achieved by one of the eventual versions of the product over time. Five maturity levels are defined in the order of increasing liability risk:\r\n \r\n\n1. Design - There is only a theoretical design that is still to be fully developed. No liability of the product originator applies.\r\n\n2. Prototype - The early design phases have been completed and the first functional prototype has been built. No liability of the product originator applies.\r\n\n3. Production\/DIY - The product is fully defined and documented and can be replicated. No liability of the product originator applies.\r\n\n4. Production\/Kit - The product is sold by a commercial actor as a kit. Limited availability applies to the vendor.\r\n\n5. Production\/full product - The product is sold by a company as a finished product. Full availability applies to the vendor.\n\n\n\nl\n\nStatus of the community\n\nBinary value indicating whether the community is active. The community is considered inactive in case no activity (encompassing either product development or sales and marketing) can be detected on the website or the collaboration platforms within one year.\n\n\nm\n\nProduct category\n\nClassification of the products in product groups\n\n\n\nCriteria are evaluated based on information that can be accessed through the websites of the product development communities. For the four criteria \u201cCAD files available,\u201d \u201cassembly instructions available,\u201d \u201cbill of materials available,\u201d and \u201cguidelines for participation,\u201d the following practical cut-off rule is applied for each binary criterion: If the necessary information to satisfy a criterion cannot be found in less than 10 minutes, it is considered unavailable. This cut-off rule is consistent with the fact that accessibility to documents required by the Open Source Definition implies not only that these documents can be accessed, but also that they can be found easily.\nNote that with the focus of the prior subsection titled \u201cFocus: Complex non-electronic hardware products,\u201d solely the documentation relative to the non-electronic hardware has been assessed. Documentation regarding electronic hardware and software has not been regarded. The authors emphasize here that this does not mean that the openness of electronic hardware has lower relevance in general. This methodological choice is instead dictated by the background of the authors and the willingness to limit volume of data to be gathered in this study. This attempt is based on the assumption that data gathered for non-electronic hardware are representative for data about electronic hardware.\nAlso, this article specifically focuses on OSH development. The use of OSH may require pieces of documentation which are not considered here, such as operating instructions and parts specifications. Withholding this information may render a product useless or even unsafe. The provision of operating instructions is for this reason legally requested in specific product branches such as an automobile. As a consequence, the provision of operational instructions has not been considered as a distinctive characteristic of OSH in this article.\n\nOpenness assessment criteria \nBased on the data defined in previous sections, four composite criteria are defined for assessing the compliance of the product-related documentation to the four factors of openness:\n\n The transparency (T) criterion addresses the freedom to study. A product satisfies this criterion when CAD files are published.\n The accessibility (A) criterion addresses the freedom to modify. A product satisfies this criterion when all published content is editable or when a guideline for participation is available.\n The replicability (R) criterion addresses the freedom to make. A product satisfies this criterion when assembly instructions and bill of materials are available.\n The commercial usability (C) criterion addresses the freedom to distribute. A product satisfies this criterion when licences applied to the non-electronic hardware allow commercial usage of the published content.\nThe four criteria are defined as logical operations on the Boolean values [a,\u2026,h] as depicted by the following equation:\n\r\n\n\n \n \n \n \n {\n \n \n \n \n \n T\n =\n a\n \n \n \n \n \n \n A\n =\n \n (\n \n \n (\n \n a\n ∨\n c\n ∨\n e\n \n )\n \n ∧\n \n (\n \n a\n →\n b\n \n )\n \n ∧\n \n (\n \n c\n →\n d\n \n )\n \n ∧\n \n (\n \n e\n →\n f\n \n )\n \n \n )\n \n ∨\n g\n \n \n \n \n \n \n R\n =\n c\n ∧\n e\n \n \n \n \n \n \n C\n =\n h\n \n \n \n \n \n \n \n \n \n {\\displaystyle \\left\\{{\\begin{array}{l}{T=a}\\\\{A=\\left({\\left({a\\vee c\\vee e}\\right)\\land \\left({a\\rightarrow b}\\right)\\land \\left({c\\rightarrow d}\\right)\\land \\left({e\\rightarrow f}\\right)}\\right)\\vee g}\\\\{R=c\\land e}\\\\{C=h}\\\\\\end{array}}\\right.}\n \n \n\r\n\nIn addition to this, a global openness index (OI) is defined as a cumulative point system. A product gets one point each time one of the criteria a, b, c, d, e, f, g, and h is satisfied. An OI = 8 means the product fully satisfies the best practices of OSH. An OI = 0 means the product satisfies none of the criteria defining OSH. The openness index is defined in the next equation:\n\r\n\n\n \n \n \n \n \n \n \n \n O\n I\n =\n \n \n \n \n Γ\n \n (\n a\n )\n \n +\n Γ\n \n (\n b\n )\n \n +\n Γ\n \n (\n c\n )\n \n +\n Γ\n \n (\n d\n )\n \n +\n Γ\n \n (\n e\n )\n \n \n \n \n \n \n \n \n +\n Γ\n \n (\n f\n )\n \n +\n Γ\n \n (\n g\n )\n \n +\n Γ\n \n (\n h\n )\n \n \n \n \n \n \n \n \n \n \n w\n h\n e\n r\n e\n  \n Γ\n \n (\n x\n )\n \n =\n \n {\n \n \n \n \n \n 1\n  \n i\n f\n  \n x\n \n \n \n \n \n \n 0\n  \n i\n f\n ¬\n  \n x\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n {\\displaystyle {\\begin{array}{ll}{OI=}&{\\Gamma \\left(a\\right)+\\Gamma \\left(b\\right)+\\Gamma \\left(c\\right)+\\Gamma \\left(d\\right)+\\Gamma \\left(e\\right)}\\\\&{+\\Gamma \\left(f\\right)+\\Gamma \\left(g\\right)+\\Gamma \\left(h\\right)}\\\\&{\\quad {where\\ \\Gamma \\left(x\\right)=\\left\\{{\\begin{array}{l}{1\\ if\\ x}\\\\{0\\ if\\neg \\ x}\\\\\\end{array}}\\right.}}\\\\\\end{array}}}\n \n \n\r\n\n\nClustering \nIn order to identify typical patterns in the publication of product-related documentation, products are clustered according to the values of the eight binary criteria [a,\u2026,h] defined above. The k-medioids algorithm (or PAM, Partitioning Around Medioids) has been used since it is particularly adapted to the clustering of objects described by categorical data. This method requires as input an n*n matrix giving a measure of the similarity of each pair of product. Each product is represented by a binary word of eight bits, with each bit representing a criterion from a to h, as described in the following equation:\n\r\n\n\n \n \n \n \n P\n \n i\n \n \n =\n \n a\n \n i\n \n \n ,\n …\n ,\n \n h\n \n i\n \n \n \n \n {\\displaystyle P_{i}=a_{i},\\ldots ,h_{i}}\n \n \nWhere:\n\n Pi is an eight-bit word representing the product i; and\n ai, \u2026, hi are the binary values of criteria a to h evaluated for product i.\n\r\n\nInter-product similarity is computed according to the Manhattan distance[44] of the products\u2019 descriptive 8-bit words as described in the following equation:\n\r\n\n\n \n \n \n \n S\n \n i\n j\n \n \n =\n \n ∑\n \n k\n =\n 1\n \n \n 8\n \n \n \n Γ\n \n (\n \n \n (\n \n \n P\n \n i\n ,\n k\n \n \n ∧\n ¬\n \n P\n \n j\n ,\n k\n \n \n \n )\n \n ∨\n \n (\n \n ¬\n \n P\n \n i\n ,\n k\n \n \n ∧\n \n P\n \n j\n ,\n k\n \n \n \n )\n \n \n )\n \n \n \n \n {\\displaystyle S_{ij}=\\sum \\limits _{k=1}^{8}{\\Gamma \\left({\\left({P_{i,k}\\land \\neg P_{j,k}}\\right)\\vee \\left({\\neg P_{i,k}\\land P_{j,k}}\\right)}\\right)}}\n \n \nWhere:\n\n Sij is the Manhattan distance of binary words Pi and Pj and is a numerical of [0, 8]. A distance of 0 means words are equal. A distance of 8 means words are completely different.\n Pi,k and Pj,k are respectively the kth bit of words Pi and Pj; and\n \u0393 is the function defined in the second equation.\n\r\n\nThe k-medioids algorithm requires also as parameter the number of clusters to be defined. The optimal number of clusters\u2014so that adding another cluster does not provide further information\u2014is calculated with the help of the elbow method.[45]\n\nResults \nA data acquisition campaign was carried out between March 2016 and April 2017. This large time period ensured sufficient exhaustiveness of the snow-ball research. At the same time, it generated risks of data obsolescence. Therefore, the gathered data set was screened and actualized in its entirety in a last verification pass performed May 2017. In total, 132 products were found which satisfied the conservative selection criteria. Further modifications of the considered objects of research that occurred after May 2017 were not considered and are not reported in this article.\n\nPool of gathered products \nThe selected products cover a wide range of categories. The largest categories represented are machine tools (33 products), vehicles (18 products), and robotics (11 products), as well as medical and laboratory equipment (9 products each, Figure 2a). The category of machine tools includes mainly desktop machine tools such as the 3D printer Ultimaker or the laser cutter Lasersaur. The vehicles category mainly consists of bikes like XYZ Space Frame Vehicles and cars like the Tabby OSVehicle. The robotics category is largely represented by humanoid robots developed for teaching or research purposes, such as the igus Humanoid Open Platform. Finally, the medical equipment category covers diverse products like OpenBionics\u2019 Prosthetic Hand or diagnostic equipment such as the echo-stethoscope echOpen.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Characterization per product category (a), per technology (b) and per project status (c)\n\n\n\nIn total, a share of 43% of these (57 products) consist exclusively of non-electronic components. The remaining share of 57% (75 products) also comprise electronic hardware as well as software (Figure 2b). Three quarters of the products are currently developed and\/or marketed; while other products originate from projects which are not active anymore (Figure 2b).\nThe distribution of products per maturity level, depicted in Figure 3, indicates that the majority of the selected products has reached a stage of development that enables systematic replication and production; 70% are in production stages.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Distribution of products per maturity level\n\n\n\nOverall openness of the published documentation \nFigure 4 depicts the number of products satisfying the criteria a to h, as well as the distribution of products along the openness index (OI). The most satisfied criterion is the publication of CAD files (criterion a, with 77%, 102\/132) and the least satisfied are the publication of a call for contribution and the editability of assembly instructions (criterion g and d, with respectively 26% and 25%, 34\/ and 33\/132). In total, 11 products satisfy all criteria [a,\u2026,h], while 10 products satisfy none of them. The arithmetic mean of the openness index of all products is 4.2 points (median value 4 points). \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Number of products satisfying the criteria a to h\n\n\n\nFigure 5 shows the number of products satisfying the four criteria of openness (T, A, R, and C). The most satisfied criterion is transparency (T, 77%, 102\/132), respectively followed by commercial usability (C, 72%, 95\/132), replicability (R, 60%, 80\/132), and accessibility (A, 41%, 54\/132). In total, 22 products comply with all the four criteria, while 11 others fulfill none of them. The average number of criteria fulfilled is 2.5 (median value 3).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Number of products satisfying the criteria T, A, R, and C\n\n\n\nThis data highlights a certain heterogeneity of practices in the publication of OSH product documentation. The large spectrum of openness observed tends to indicate that the different factors of openness are considered by practitioners more as optional factors rather than mandatory ones. On one side of the spectrum, a few products can be considered as fully open, because they satisfy the four criteria of openness (17%, 22\/132) or get the maximal openness index (11%, 15\/132). On the other side of the spectrum, another few products can be considered as fully closed, as they either do not satisfy any of the four criteria of openness (8%, 10\/132) or get a null openness index (8%, 11\/132). Between those extremes, the large majority of products show diverse profiles of partial openness, without clearly identifiable patterns.\n\nRelative frequency of openness factors \nResults provided by Figure 4 suggest that transparency and commercial usability may be aspects which are given high importance on average or which are easily achievable, while accessibility is an aspect which may be given low importance on average or difficult to achieve.\nTransparency is not associated with high additional costs, as it is related to the publication of files which are required in the development process (CAD files and schematics). Nonetheless, a significant number of products (almost one fourth) are not providing these files publicly.\nCommercial usability is related to IP risks. The commercial usability aspect is at the core of the concept of open source, yet more than one third of the products are not provided with a licence allowing commercial usage or are published under licences excluding commercial usage. In the latter case, these products are actually unambiguously not complying with the Open Source Hardware Statement of Principles 1.0, which requires explicitly the use of licences allowing commercial usability.\nThe lower occurrence rate of replicability may be explained by the corresponding additional effort. Indeed, it requires more resources to make a product replicable than transparent or commercially usable. Furthermore, while transparency is about sharing files which are required for the development process of the product and exist whether or not the product is open source, replicability requires the formalization of assembly instructions and bill of materials, which is a time intensive activity. Not all communities may be willing to make the effort to formalize these documents. Moreover, assembly instructions may only be relevant for products which are meant to be produced in a DIY production setting. In the context of projects dedicated to the development of complex OSH products which are meant for industrial production, assembly instructions may be secondary or even irrelevant information.\nThe lower occurrence rate of accessibility may also be explained by the corresponding additional effort. As introduced earlier, accessibility is a precondition for the emergence of a community-based co-development process. Integrating contributions from a development community implies significant coordination costs and requires dedicated resources. As accessibility supports a process which is difficult to implement in practice, not all product originators may give this aspect a high priority.\n\nRole of contextual criteria in openness \nTable 2 compares the openness of all products categorized along three assessed contextual criteria: product maturity, whether products are purely mechanical or mechatronic, and whether the surrounding community is active or inactive.\n\n\n\n\n\n\n\nTable 2. Openness vs. contextual criteria\n\n\n\n\nNumber of products\n\nOI\n\nTransparency\n\nAccessibility\n\nReplicability\n\nCommercial usability\n\n\n\n\n\n\nMean value\n\nAbsolute\n\nRelative\n\nAbsolute\n\nRelative\n\nAbsolute\n\nRelative\n\nAbsolute\n\nRelative\n\n\nConcept\n\n14\n\n1.64\n\n4\n\n29%\n\n6\n\n43%\n\n1\n\n7%\n\n7\n\n50%\n\n\nPrototype\n\n26\n\n2.54\n\n14\n\n54%\n\n10\n\n38%\n\n6\n\n23%\n\n16\n\n62%\n\n\nProduction\/DIY\n\n61\n\n4.93\n\n54\n\n89%\n\n26\n\n43%\n\n48\n\n79%\n\n46\n\n75%\n\n\nProduction\/kit\n\n16\n\n5.38\n\n15\n\n94%\n\n4\n\n25%\n\n15\n\n94%\n\n14\n\n88%\n\n\nProduction\/full product\n\n15\n\n4.80\n\n15\n\n100%\n\n8\n\n53%\n\n10\n\n67%\n\n12\n\n80%\n\n\nMechanic\n\n57\n\n3.65\n\n38\n\n67%\n\n25\n\n44%\n\n30\n\n53%\n\n39\n\n68%\n\n\nMechatronic\n\n75\n\n4.53\n\n64\n\n85%\n\n29\n\n39%\n\n50\n\n67%\n\n56\n\n75%\n\n\nActive\n\n99\n\n4.57\n\n83\n\n84%\n\n46\n\n46%\n\n67\n\n68%\n\n76\n\n77%\n\n\nInactive\n\n33\n\n2.91\n\n19\n\n58%\n\n8\n\n24%\n\n13\n\n39%\n\n19\n\n58%\n\n\n\nA first possible reason for the absence of published documentation may be that this documentation does not exist and cannot exist in the level of detail required by the criterion used here. Indeed, in early design stages, the product documentation may not be mature enough for the product concept to be formalized into CAD files, assembly instructions, and bills of materials. This explanation tends to be supported by the gathered data, as the average openness index of products in the early phases (concept and prototype) is significantly lower than those of production phases (DIY, kit, full product production). The percentage of products satisfying the criterion of transparency grows along with product maturity: it is lower than 54% in the concept and prototype phases and grows over 89% in the other phases. The average openness index increases as well; however, it starts to sink again at the last stage (full product production) due to lower replicability and commercial reusability. In this phase, products are marketed by companies which are taking financial risks. This may make them hesitant to disclose information that facilitates imitation.\nA second possible reason for the absence of published documentation may be that the community which drove the product development is not active anymore. In that case, there is not enough workforce anymore to maintain data online; links become \u201cdead links\u201d and information finally disappears. This explanation tends to be supported by the gathered data, as the average openness index is significantly higher for products with active surrounding communities than for products with inactive communities (respectively 4.57 points and 2.91 points).\nA third possible reason for the absence of published documentation may be that the community which drives the product development does not follow an open-source approach for the entire product but only for selected components. A straightforward case is when products include off-the-shelf components which are not open source. Another case is when originators deliberately exclude a developed component from the open-source approach in order to protect their strategic knowledge. This behavior has been previously empirically observed and is discussed by Balka, Raasch, and Herstatt.[5] It is however explicitly excluded by the Open Source Hardware Certification Programme of the Open Source Hardware Association, as it is claimed that \u201call parts, designs, code, and rights under the control of the creator must be made open.\u201d[46] In the current study, only the openness of mechanical components has been assessed, and those of software and electronic hardware components contained in mechatronic products have been left aside. It is therefore possible that a mechatronic product appears here as non-open although the corresponding electronic hardware and software are open source. Following this explanation, there would be a high chance that the average openness of mechanical parts of mechatronic products should be lower than those of purely mechanical products. This explanation is however not supported by the gathered data, as the average openness index is higher for mechatronic products than for purely mechanical products.\nThe following other possible interpretations of the absence of published documentation cannot be verified by the data gathered in this article:\n\n The intention of openness is given in the product development project, but the project lacks capacity to document and to put the product-related documentation online. Although in open-source communities publishing documentation is seen as a way to gather new workforce, generating product documentation remains a time-consuming process and may get lower priority than other product development activities.\n There is a certain delay between the statement that the product is open source and the actual disclosure of product-related information. Project initiators may start claiming a product is open source before the corresponding documentation is put online.\n The product is claimed to be open source whereas no intention to publish product-related information is given in the product development project, what we can term here as \u201copenwashing\u201d in analogy to \u201cgreenwashing\u201d or \u201cwhitewashing.\u201d\nTypical profiles \nIn order to identify possible typical profiles, the k-medioids algorithm has been computed on the dataset leading to the identification of five clusters. The similarity matrix reordered according to the identified clusters is displayed in Figure 6. \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. Heatmap representing the Manhattan distance Sij between each pair of the 132 products (dark green pairs are similar, light green are dissimilar)\n\n\n\nThe openness of the products of each cluster is displayed in Table 3 and described hereafter:\n\n Cluster 1. Fully open products which satisfy almost all openness criteria and having hence a high average OI value (mean value 6.83\/8)\n Cluster 2 and 3. Products being transparent, replicable and commercially usable but not accessible; their average OI value is medium to high (mean value respectively 5.23 and 3.81\/8) and is mainly handicapped by the lack of accessibility.\n Cluster 4. Products being transparent, accessible and commercially usable but not replicable; their average OI value is medium to low (mean value 2.94\/8) and is mainly handicapped by the lack of replicability\n Cluster 5. Fully closed products, i.e., products for which almost no documentation can be found and out of which two thirds do not provide commercially usable documentation; their average OI value is very low (mean value 0.83\/8)\n\n\n\n\n\n\nTable 3. Openness of the four identified clusters\n\n\nCluster\n\nNumber of products\n\nOI\n\nTransparency\n\nAccessibility\n\nReplicability\n\nCommercial usability\n\n\n\n\n\n\nMean value\n\nAbsolute\n\nRelative\n\nAbsolute\n\nRelative\n\nAbsolute\n\nRelative\n\nAbsolute\n\nRelative\n\n\nC1\n\n29\n\n6.83\n\n26\n\n90%\n\n24\n\n83%\n\n28\n\n97%\n\n24\n\n83%\n\n\nC2\n\n31\n\n5.23\n\n31\n\n100%\n\n7\n\n235\n\n27\n\n87%\n\n26\n\n84%\n\n\nC3\n\n31\n\n3.81\n\n28\n\n90%\n\n5\n\n16%\n\n25\n\n81%\n\n24\n\n77%\n\n\nC4\n\n17\n\n2.94\n\n15\n\n88%\n\n10\n\n59%\n\n0\n\n0%\n\n13\n\n76%\n\n\nC5\n\n24\n\n0.83\n\n2\n\n8%\n\n8\n\n33%\n\n0\n\n0%\n\n8\n\n33%\n\n\n\nApart from products showing full or near zero openness (C1 and C5), this clustering displays two other profiles: products developed in the frame of projects thriving at all openness aspects excluding accessibility (C2 and C3), and products developed in the frame of projects thriving at all openness aspects excluding replicability (C4). These results tend to highlight the existence of two OSH project archetypes: the one using openness as a way to support the emergence of collaborative development in communities, the other using openness as a way to support the broad diffusion of the outcome of a conventional and closed product development process. Cluster C1 can be interpreted as a combination of these two archetypes, i.e., as projects of C4 which achieved the product development phase, reached the production and commercialization phase and started to strive for replicability.\n\nDiscussion \nWhat emerges from the analysed data is a multifaceted picture of OSH which confirms the existence of a \u201cconfusion on what actually makes a project an open source project\u201d identified by Gacek and Arief[34] in the field of OSS. Product originators tend to make use of the large room for interpretation given by the fuzzy definition of the term \u201copen source hardware,\u201d to pick up the openness factors best fitting with their situation and therefore to answer for themselves the question \u201cwhat is the source of open source hardware?\u201d in different ways.\nWhereas none of the four openness factors are represented by more than three fourths of the gathered products, the factors of transparency and commercial usability are nonetheless considered mandatory by most practitioners. The respective non-compliance rates of 23% and 28% may be considered as a normal and reasonable gap between theory and practice. Transparency is the most represented openness factor, and the non-compliance to this factor can be largely explained by the fact that some products are not mature enough for CAD files to be produced and published. Commercial usability can simply be considered as mandatory as it is the only factor of openness explicitly addressed in both the Open Source Definition and the Open Source Hardware Statement of Principles 1.0. Furthermore, it can be viewed as a prerequisite to flank transparency.\nThe somewhat optional nature of the openness factors of accessibility and replicability bases the identification of two main product development project archetypes in OSH. One makes use of open-source publication of product-related documentation as a means to support community-based product development, while the other makes use of these same means for supporting diffusion of their privately developed product. Interestingly, projects of the first group are less numerous than those of the second group. These results are in contradiction with the general use of the term \u201copen source\u201d as a depiction of a collaborative product development process (as stated by Gacek and Arief[34]).\n\nFraming the \u201csource\u201d of open source hardware \nIt appears from the reported analysis that the \u201csource\u201d of OSH tends to be interpreted in practice as a dataset whose contents and properties evolve along the product lifecycle. This is summarized in the open-source hardware lifecycle depicted in Figure 7. This original framework defines two states of OSH, characterized by specific motivations and shared product documentation as well as two modes for developing open source hardware products. It also specifies in more detail the concepts which contours have been previously defined in scholarly contributions.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 7. Open-source hardware lifecycle\n\n\nWithin the product development process, an open-source approach may be used as a means to support the emergence of community-based product development. The OSH product is then before all the object of an open-source product development process (state 1 in Figure 7). In order to be labelled as \u201copen source,\u201d this process requires at least commercial usability, transparency, and accessibility. At early development stages, transparency can only be realized through the publication of descriptive text and simple schematics. Along the development of the product, stable technical drawings emerge (CAD files) which have to be shared in order to support further collaborative development. Accessibility is ensured by the use of editable formats (e.g., native and open CAD formats instead of STL files, native and open text editor files instead of PDF files).\nOnce a product is fully defined and can be produced, it can be documented as an open-source product (state 2 in Figure 7). In this case, open source is used to support the production and diffusion of the product and its durability (through repairability, modifiability and upgradability). In order to be labelled as \u201copen source,\u201d this product can be delivered with additional information, such as bill of materials and assembly instructions, which support production. Making use of this information for actual production and commercialization purposes requires this content to be published with licences allowing commercial usage. During the usage phase and at the product end-of-life, access to this same information may support the aspects of durability mentioned above.\nAn open source product can either be the result of an open-source product development process (mode 1 in Figure 7) or of the disclosing of documentation developed in a private setting\u2014defined as public innovation (mode 2 in Figure 7) by Huizingh.[32] Note that the latter is not an open-source development approach because it neither supports accessibility nor transparency. Upon the release of an open-source product, the basis is made for new versions to be developed within any one of the two modes. Once a new version of an open-source product is released, it becomes the basis for further continuous open-source product development or public innovation within a private innovation process.\nThe state \u201copen-source product development\u201d has been previously termed in literature as open-source innovation[32][14] and as open design.[35][31] The state \u201copen-source product\u201d is the result of either open-source innovation or public innovation as termed by Huizingh.[32] The lifecycle model is consequently in contradiction to an exclusive definition of OSH as a subcategory of open-source innovation, as defined by Raasch, Herstatt, and Balka[14] as well as Huizingh.[32] This would omit a large part of the actual practices rightly claiming the label of \u201copen-source hardware,\u201d which also includes the free revealing of innovations achieved in private settings. In those cases, however, labeling the product as open source would only make sense after a public release. An open-source product development project as defined here may already fulfil the necessary requirements earlier.\n\nLimits of the reported research \nThe list of OSH products gathered for this study cannot be claimed to be exhaustive, and its representativeness for the entire field of OSH has to be considered cautiously. The authors also cannot exclude that the method used to search for eligible products may have omitted subsets of the field. Particularly, product development projects which are in the early development phases may be more difficult to find because they are still poorly documented and their communities are relatively small, if there is any at all yet. This may be the same for projects which acquired low publicity and are therefore poorly ranked in common internet search engines. Nonetheless, the list of OSH products gathered for this study, with this focus, is the largest that has been published so far. At the time the data acquisition campaign was terminated (April 2017), the marginal search time for finding a new project was high enough to justify freezing the dataset since the risk of omitting a significant part of the studied phenomenon was reasonably low.\nAdditionally, the broad field of OSH was narrowed down to discrete, tangible, non-electronic and complex products, which excludes a large part of the field (such as millions of gimmicks). In terms of how far the results discussed in this article are valid for the whole range of OSH products, including low complexity products as well as purely electronic products, remains an open question. Further research could make use of the methods defined in this article in order to perform a similar analysis specifically addressing electronic hardware. To the knowledge of the authors, no such study has been published so far. The OSHWA maintains a list of certified OSH products, most of which are electronic products, providing therefore an interesting data basis.\nThe assessment of the corresponding published product-related documentation has been simplified to the evaluation of binary criteria. Each product was assessed as to whether certain types of documents are provided. The quality of the published documentation with regard to the level of detail, comprehensiveness, or clarity has not been examined. Do published CAD files represent the whole product or just parts of it? Are the guidelines for participation easily understandable for potential participants and provide them with the right information? What information is displayed in parts lists and in what form? These questions were not taken into account because of the need to reduce the data acquisition effort to a manageable level. This simplification may have produced a positive bias, whereby products may have been rated more open than they are. Future research is needed in order to define to what extent published product-related documentation is actually usable and useful.\nOn the other hand, the practical cut-off rule for each binary criterion may have introduced a negative bias. Products may have been rated less open than they are. Simply because some pieces of documentation have not been found after 10 minutes of internet search does not necessarily mean that information cannot be found online. The cut-off rule applied is a practical interpretation of the Open Source Definition which requires that documents can be accessed \u201ceasily.\u201d Another negative bias comes from the fact that only information related to the non-electronic hardware components included in the respective products was evaluated. Documentation regarding electronic hardware and software has not been regarded as a practical cut-off in data acquisition (see the subsection on \u201cData acquisition\u201d), hence, parts of openness have been omitted. Needless to say, the openness of electronic hardware is as important as those of other types of hardware. Consequently, the openness of a mechatronic product should be assessed upon those of its electronic and mechanical components.\nThe dataset on which this article is based upon has been published with a CC-BY licence.[47] An actualized version of this dataset is available under the same licensing terms in the Open Source Hardware Directory. This database of complex OSH products allows any interested person to add references of OSH products, or to edit existing ones. The content of this database can be used for any purpose, such as future research, as suggested in this article. The objective of the methodology pursued in this article and in the generation of the data was to generate an overview of the field of OSH. The authors cannot guarantee that the presented data of more than 2000 items is free of errors. Should the originators or contributors to the OSH products referenced in this dataset consider their products misrepresented, the authors kindly invite them to make corrections. They are free to edit the provided database or inform the authors of detected errors.\n\nConclusion \nThis article provides an overview of how the concept of OSH is interpreted in practice, in other words, what is the \u201csource\u201d of OSH for practitioners. Starting from a critical analysis of existing definitions, evaluation criteria have been defined and used to analyse the published documentation of a pool of 132 complex and non-electronic OSH products.\nThe gathered data illustrates how the field of OSH has moved well beyond the scope of electronic hardware and towards the field of non-electronic complex product development. The reported research required the establishment of a database of OSH products, which is the largest published to date. This database provides empirical evidence of the evolution of OSH outside the sphere of electronics, a phenomenon which presently has only been described by scholarly publications based on isolated cases.\nThe analysis of the published documentation of the 132 identified OSH products confirms the wide range of documentation sharing practices which may also be due to the fuzzy definitions of OSH. Product documentation ranges from very demanding to very lax compliance, i.e., from the mere sharing of CAD files to the publication of comprehensive sets of documents. The results highlight a surprisingly large share of products which are provided with barely any information at all (cluster C5, 18%). This could either be due to a misinterpretation of the concept of OSH, a deliberate intention to \u201copenwash\u201d a product, or the influence of the time variable. What can be understood from this is that there is a need to set standards in order to achieve clarity in the field of OSH. This will support the emergence of a constructive public discourse around these new practices.\nClustering the characteristics of published product-related documentation led to the identification of two distinct states of OSH. These have been understood as resulting from differing though not mutually exclusive strategic modes: namely the stimulation of community-based product development and the public dissemination of privately achieved innovation. The interaction between those states has been summarized in the definition of an OSH lifecycle summarizing observed approaches to OSH. It is the hope of the authors that this article has introduced solid categories that have provided a foundation on which further practice and research activities can be based.\n\nFootnotes \n\n\n\u2191 For an overview of OSH licences, see for example Katz.[6] \n\n\u2191 Thingiverse is a platform that allows users to share a huge variety of 3D models based on CAD or STL files. \n\n\u2191 Enable Outreach is a network of makers that collaborate locally with disabled children to provide them with adapted 3D printed hand prosthesis. \n\n\u2191 Evolutionary design may not generally contradict parallelization or staging of processes, i.e., in the intra-project environment for scaling and efficiency. Also, this is different from the predetermined mechanism of breaking down problems into constituent parts (modularization) and avoids creation of task interdependencies. \n\n\u2191 These four freedoms are derived from the free software definition[26], which are: Freedom 0, the freedom to run the program for any purpose; Freedom 1, the freedom to study how the program works; Freedom 2, the freedom to redistribute copies; Freedom 3, the freedom to distribute copies of modified versions. In the transition from the context of immaterial intellectual property to the realm of tangible products, these freedoms have been reinterpreted. For example, running a program requires compiling the source code (an action alternatively termed as build or make in the software jargon). The freedom to run became the freedom to make the product, that is, to produce it. \n\n\u2191 or deeper insights into the different degrees of involvement in open source communities, see for example Gacek and Arief.[34] \n\n\nAcknowledgements \nThe reported research has been performed in the frame of the French-German interdisciplinary research project \u201cOpen! \u2013 Methods and tools for community-based product development.\u201d It is jointly funded by the French and German national science agencies ANR (Agence Nationale de la Recherche, grant ANR-15-CE26-0012) and DFG (Deutsche Forschungsgemeinschaft, grants JO 827\/8-1 and STA 1112\/13-1). The authors would like thank the anonymous reviewers for their insightful suggestions and constructive feedback. Last but not least, they would like to acknowledge Kerstin Carola Schmidt for her contributions to the data collection and assessment.\n\nCompeting interests \nThe authors have no competing interests to declare.\n\nReferences \n\n\n\u2191 Hatch, M. (2013). The Maker Movement Manifesto: Rules for Innovation in the New World of Crafters, Hackers, and Tinkerers. McGraw-Hill Education. ISBN 9780071821124.   \n\n\u2191 Voigt, C.; Montero, C.S.; Menichinelli, M. (2016). \"An Empirically Informed Taxonomy for the Maker Movement\". In Bagnoli, F., Satsiou, A.; Stavrakakis, I. et al.. Internet Science - INSCI 2016. Lecture Notes in Computer Science. 9934. Springer. doi:10.1007\/978-3-319-45982-0_17. ISBN 9783319459820.   \n\n\u2191 Balka, K. (2011). Open Source Product Development - The Meaning an Relevance of Openness. Gabler Verlag. p. 4. doi:10.1007\/978-3-8349-6949-1. ISBN 9783834969491.   \n\n\u2191 4.0 4.1 4.2 Raasch, C. (2011). \"Product Development in Open Design Communities: A Process Perspective\". International Journal of Innovation and Technology Management 8 (4): 557\u201375. doi:10.1142\/S021987701100260X.   \n\n\u2191 5.0 5.1 5.2 5.3 5.4 5.5 Balka, K.; Raasch, C.; Herstatt, C. (2014). \"The Effect of Selective Openness on Value Creation in User Innovation Communities\". The Journal of Product Innovation Management 31 (2): 392\u2013407. doi:10.1111\/jpim.12102.   \n\n\u2191 Katz, A. (2012). \"Towards a Functional Licence for Open Hardware\". The Journal of Open Law, Technology & Society 4 (1): 41\u201362. doi:10.5033\/ifosslr.v4i1.69.   \n\n\u2191 Balka, K. (2016). \"Project List\". Open Innovation Projects. Archived from the original on 27 September 2016. https:\/\/web.archive.org\/web\/20160927222446\/http:\/\/open-innovation-projects.org\/project-list .   \n\n\u2191 Moritz, M.; Redlich, T.; Krenz, P. et al. (2015). \"Open up or Close down - The new Era of \u201cOpenneers\u201d and how they lead the Way to Future Success\". Journal of Systemics, Cybernetics and Informatics 13 (6): 15\u201322. http:\/\/www.iiisci.org\/journal\/sci\/FullText.asp?var=&id=HB188KY15 .   \n\n\u2191 9.0 9.1 Pearce, J.M. (2017). \"Emerging Business Models for Open Source Hardware\". Journal of Open Hardware 1 (1): 2. doi:10.5334\/joh.4.   \n\n\u2191 10.0 10.1 Li, Z.; Seering, W.; Ramos, J.D. et al. (2017). \"Why Open Source?: Exploring the Motivations of Using an Open Model for Hardware Development\". Proceedings of the ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. doi:10.1115\/DETC2017-68195.   \n\n\u2191 Fjeldsted, A.S.; Adalsteinsdottir, G.; Howard, T.J. et al. (2012). \"Open Source Development of Tangible Products - From a business perspective\". Proceedings of NordDesign 2012. https:\/\/orbit.dtu.dk\/en\/publications\/open-source-development-of-tangible-products-from-a-business-pers .   \n\n\u2191 Bonvoisin, J.; Boujut, J.-F. (2015). \"Open Design Platforms for Open Source Product Development: Current State and Requirements\". Proceedings of the 20th International Conference on Engineering Design 8: 11\u201320. https:\/\/www.designsociety.org\/publication\/37899\/OPEN+DESIGN+PLATFORMS+FOR+OPEN+SOURCE+PRODUCT+DEVELOPMENT%3A+CURRENT+STATE+AND+REQUIREMENTS .   \n\n\u2191 13.0 13.1 Howard, T.J.; Achiche, S.; \u00d6zkil, A. et al. (2012). \"Open Design and Crowdsourcing: Maturity, Methodology and Business Models\". Proceedings of DESIGN 2012: 181\u201390. https:\/\/www.designsociety.org\/publication\/31985\/OPEN+DESIGN+AND+CROWDSOURCING%3A+MATURITY%2C+METHODOLOGY+AND+BUSINESS+MODELS .   \n\n\u2191 14.0 14.1 14.2 14.3 Raasch, C.; Herstatt, C.; Balka, K. (2009). \"On the open design of tangible goods\". R&D Management 39 (4): 382\u201393. doi:10.1111\/j.1467-9310.2009.00567.x.   \n\n\u2191 Hansen, A.; Howard, T.J. (2013). \"The Current State of Open Source Hardware: The Need for an Open Source Development Platform\". ICoRD'13: 977\u201388. doi:10.1007\/978-81-322-1050-4_77.   \n\n\u2191 Camarinha,-Matos, L.M.; Afsarmanesh, H.; Galeano, N. et al. (2009). \"Collaborative networked organizations \u2013 Concepts and practice in manufacturing enterprises\". Computers & Industrial Engineering 57 }issue=1: 46\u201360. doi:10.1016\/j.cie.2008.11.024.   \n\n\u2191 Maher, M.L.; Paulini, M.; Murty, P. (2010). \"Scaling Up: From Individual Design to Collaborative Design to Collective Design\". Proceedings from Design Computing and Cognition '10: 581\u201399. doi:10.1007\/978-94-007-0510-4_31.   \n\n\u2191 Paulini, M.; Murty, P.; Maher, M.L. (2013). \"Design processes in collective innovation communities: A study of communication\". CoDesign - International Journal of CoCreation in Design and the Arts 9 (2): 90\u2013112. doi:10.1080\/15710882.2012.716850.   \n\n\u2191 West, J.; Lakhani, K.R. (2008). \"Getting Clear About Communities in Open Innovation\". Industry and Innovation: 223\u201331. doi:10.1080\/13662710802033734.   \n\n\u2191 von Hippel, E. (2006). Democratizing Innovation. MIT Press. pp. 4, 165. ISBN 9780262720472.   \n\n\u2191 Aksulu, A.; Wade, M.R. (2010). \"A Comprehensive Review and Synthesis of Open Source Research\". Journal of the Association for Information Systems 11 (11): 576\u2013656. doi:10.17705\/1jais.00245.   \n\n\u2191 Lee, G.K.; Cole, R.E. (2003). \"From a Firm-Based to a Community-Based Model of Knowledge Creation: The Case of the Linux Kernel Development\". Organization Science 14 (6): 615\u2013758. doi:10.1287\/orsc.14.6.633.24866.   \n\n\u2191 Howison, J.; Crowston, K. (2014). \"Collaboration Through Open Superposition: A Theory of the Open Source Way\". MIS Quarterly 38 (1): 29\u201350. https:\/\/misq.org\/collaboration-through-open-superposition.html .   \n\n\u2191 Maher, M.L.; Poon, J.; Boulanger, S. (1996). \"Formalising Design Exploration as Co-Evolution\". Advances in Formal Design Methods for CAD: 3\u201330. doi:10.1007\/978-0-387-34925-1_1.   \n\n\u2191 Open Source Hardware Association (2016). \"Definition (English)\". https:\/\/www.oshwa.org\/definition\/ .   \n\n\u2191 Free Software Foundation (2015). \"What is free software?\". GNU.org. https:\/\/www.gnu.org\/philosophy\/free-sw.en.html .   \n\n\u2191 Open Source Initiative (2007). \"The Open Source Definition (Annotated)\". https:\/\/opensource.org\/osd-annotated .   \n\n\u2191 28.0 28.1 Bonvoisin, J.; Schmidt, K.C. (15 February 2017). \"Best practices of open source mechanical hardware: A guide with practical advice for sharing product-related documentation\". DepositOnce. doi:10.14279\/depositonce-5729. https:\/\/depositonce.tu-berlin.de\/handle\/11303\/6164 .   \n\n\u2191 Wang, H.; Johnson, A.L.; Bracewell, R.H. (2012). \"The retrieval of structured design rationale for the re-use of design knowledge with an integrated representation\". Advanced Engineering Informatics 26 (2): 251\u201366. doi:10.1016\/j.aei.2012.02.003.   \n\n\u2191 von Hippel, E. (2010). \"Comment on \u2018Is open innovation a field of study or a communication barrier to theory development?\u2019\". Technovation 30 (11\u201312): 555. doi:10.1016\/j.technovation.2010.09.003.   \n\n\u2191 31.0 31.1 Aitamurto, T.; Holland, D.; Hussain, S. (2015). \"The Open Paradigm in Design Research\". Design Issues 31 (4): 17\u201329. doi:10.1162\/DESI_a_00348.   \n\n\u2191 32.0 32.1 32.2 32.3 32.4 Huizingh, E.K.R.E. (2011). \"Open innovation: State of the art and future perspectives\". Technovation 31 (1): 2\u20139. doi:10.1016\/j.technovation.2010.10.002.   \n\n\u2191 33.0 33.1 33.2 33.3 Bonvoisin, J.; Laetitia, T.; Mies, R. et al. (2017). \"Current state of practices in open source product development\". Proceedings of the 21st International Conference on Engineering Design 2: 111\u2013120. https:\/\/www.designsociety.org\/publication\/39565\/Current+state+of+practices+in+open+source+product+development .   \n\n\u2191 34.0 34.1 34.2 34.3 Gacek, C.; Arief, B. (2004). \"The many meanings of open source\". IEEE Software 21 (1): 34\u201340. doi:10.1109\/MS.2004.1259206.   \n\n\u2191 35.0 35.1 35.2 Balka, K.; Raasch, C.; Herstatt, C. (2009). \"Open source enters the world of atoms: A statistical analysis of open design\". First Monday 14 (11). doi:10.5210\/fm.v14i11.2670.   \n\n\u2191 Gibb, A. (2014). Building Open Source Hardware: DIY Manufacturing for Hackers and Makers. Addison-Wesley Professional. ISBN 9780321906045.   \n\n\u2191 Jacobs, M.A. (2007). \"Product Complexity: A Definition and Impacts on Operations\". Decision Line 38 (5): 6\u20139, 21. https:\/\/ecommons.udayton.edu\/mis_fac_pub\/94 .   \n\n\u2191 Bonvoisin, J.; Galla, J.K.; Prendeville, S. (2017). \"Design Principles for Do-It-Yourself Production\". Proceedings from Sustainable Design and Manufacturing 2017: 77\u201386. doi:10.1007\/978-3-319-57078-5_8.   \n\n\u2191 39.0 39.1 Rodriguez-Toro, C.A.; Jared, G.; Swift, K. (2004). \"Product-Development Complexity Metrics: A Framework for Proactive-DFA Implementation\". Proceedings of DESIGN 2004: 483\u201390. https:\/\/www.designsociety.org\/publication\/19794\/PRODUCT-DEVELOPMENT+COMPLEXITY+METRICS%3A+A+FRAMEWORK+FOR+PROACTIVE-DFA+IMPLEMENTATION .   \n\n\u2191 Kyriakou, H.; Nickerson, J.V.; Sabnis, G. (2017). \"Knowledge Reuse for Customization: Metamodels in an Open Design Community for 3D Printing\". MIS Quarterly 41 (1): 315-332. doi:10.25300\/MISQ\/2017\/41.1.17.   \n\n\u2191 Ehls, D. (2015). \"Diversity of Participants in Open Source Projects: Comparing Individual Demographics and Participation Rationales in Software, Content, Fun, and Business Communities\". In Herstatt, C.; Ehls, D.. Open Source Innovation: The Phenomenon, Participant's Behaviour, Business Implications. Routledge. pp. 63\u201380. ISBN 9781315754482.   \n\n\u2191 Ulrich, K.T.; Eppinger, S.D. (2011). Product Design and Development (5th ed.). McGraw-Hill Education. ISBN 9780073404776.   \n\n\u2191 Open Source Hardware Association (2013). \"Best Practices for Open-Source Hardware 1.0\". https:\/\/www.oshwa.org\/sharing-best-practices\/ .   \n\n\u2191 Choi, S.-S.; Cha, S.-H.; Tappert, C.C. (2010). \"A Survey of Binary Similarity and Distance Measures\". Journal of Systemics, Cybernetics and Informatics 8 (1): 43\u201348. http:\/\/www.iiisci.org\/journal\/sci\/FullText.asp?var=&id=GS315JG .   \n\n\u2191 Tibshirani, R.; Walther, G.; Hastie, T. (2001). \"Estimating the number of clusters in a data set via the gap statistic\". Statistical Methodology Series B 63 (2): 411\u201323. doi:10.1111\/1467-9868.00293.   \n\n\u2191 Open Source Hardware Association (2017). \"Requirements for Certification\". https:\/\/certificate.oshwa.org\/ .   \n\n\u2191 Bonvoisin, J.; Schmidt, K.C. (07 July 2017). \"Openness assessment of 132 open source hardware products\". DepositOnce. doi:10.14279\/depositonce-5977. https:\/\/depositonce.tu-berlin.de\/handle\/11303\/6430 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance. To more easily differentiate footnotes from references, the original footnotes (which where numbered) were updated to use lowercase letters. Some footnotes simply referencing web pages were either turned into inline links or removed entirely for being superfluous. In the original, the citation for the Free Software Foundation 2015 is inadvertently missing; the presumed citation is added in this version.<\/blockquote>\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_%22source%22_of_open-source_hardware%3F\">https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_%22source%22_of_open-source_hardware%3F<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles (with rendered math)LIMSwiki journal articles on information technologyLIMSwiki journal articles on open source\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 14 January 2020, at 04:12.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,137 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","3ab9770e7fca9810f5573456215e3853_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_What_is_the_source_of_open-source_hardware skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:What is the \"source\" of open-source hardware?<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>What \u201copen source\u201d means once applied to tangible products has been so far mostly addressed through the light of licensing. While this approach is suitable for software, it appears to be over-simplistic for complex hardware products. Whether such a product can be labelled as open-source is not only a question of licence but a question of documentation, i.e. what is the information that sufficiently describes it? Or in other words, what is the \u201csource\u201d of open-source hardware? To date there is no simple answer to this question, leaving large room for interpretation in the usage of the term. Based on analysis of public documentation of 132 products, this paper provides an overview of how practitioners tend to interpret the concept of open-source hardware. It specifically focuses on the recent evolution of the open-source movement outside the domain of electronics and DIY to that of non-electronic and complex open-source hardware products. The empirical results strongly indicate the existence of two main usages of open-source principles in the context of tangible products: publication of product-related documentation as a means to support community-based product development and to disseminate privately developed innovations. It also underlines the high variety of interpretations and even misuses of the concept of open-source hardware. This reveals in turn that this concept may not even be clear to practitioners and calls for more narrowed down definitions of what has to be shared for a product to be called open source. This article contributes towards this effort through the definition of an open-source hardware lifecycle, summarizing the observed approaches to open-source hardware.\n<\/p><p><b>Keywords<\/b>: open-source hardware, open design, open innovation, open-source innovation, open-source product development\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>We are currently witnessing an increasing number of initiatives transferring product development and production from the private sector to the public. Enabled by the growing accessibility of affordable manufacturing technology, this is manifested in the expansion of the so-called \u201cmaker culture,\u201d which takes action to install participational production as an alternative to industrial production.<sup id=\"rdp-ebb-cite_ref-HatchTheMaker13_1-0\" class=\"reference\"><a href=\"#cite_note-HatchTheMaker13-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VoigtAnEmp16_2-0\" class=\"reference\"><a href=\"#cite_note-VoigtAnEmp16-2\">[2]<\/a><\/sup> The emergence of this culture is interwoven with the phenomenon of open-source hardware (OSH), which transfers open-source principles (as defined by Open Source Initiative 2007) from their origins in software development to the world of physical objects.<sup id=\"rdp-ebb-cite_ref-BalkaOpen11_3-0\" class=\"reference\"><a href=\"#cite_note-BalkaOpen11-3\">[3]<\/a><\/sup> While these new practices are raising significant attention, they are still in their infancy and struggle to reveal their full economic, social, and environmental potential. One of the challenges they face is that sharing knowledge about atoms is not as frictionless as sharing bits.\n<\/p><p>Both practitioners and the scientific community generally acknowledge that online sharing of a piece of hardware is more difficult than the sharing of a piece of software (e.g., see discussion of this point by Raasch<sup id=\"rdp-ebb-cite_ref-RaaschProduct11_4-0\" class=\"reference\"><a href=\"#cite_note-RaaschProduct11-4\">[4]<\/a><\/sup>. Software is digital by nature; it is made of series of characters in a format that can be shared and displayed online without specific tools, with a text editor being enough. Hardware may need to be described through more complex constructs like 2D or 3D schematics, which may require more specific software to be edited and displayed. Based on the evaluation of a pool of 20 OSH projects whose products embedded both software and hardware components, Balka, Raasch, and Herstatt<sup id=\"rdp-ebb-cite_ref-BalkaTheEffect14_5-0\" class=\"reference\"><a href=\"#cite_note-BalkaTheEffect14-5\">[5]<\/a><\/sup> highlighted that hardware components were generally less documented than the software components. This result raises questions in terms of practice. When a piece of hardware is poorly documented, is it still open source? What does \u201cless documented\u201d mean? What are the minimal requirements for labeling a hardware product as \"open source\"?\n<\/p><p>In the absence of clear guidance on this issue, it is not easy to draw a line between which piece of hardware is open source and which is not, even when licensing terms may be clear. Unlike in software, attributing appropriate licences<sup id=\"rdp-ebb-cite_ref-7\" class=\"reference\"><a href=\"#cite_note-7\">[a]<\/a><\/sup> is not sufficient to call hardware open source. Given OSH is a sociotechnical phenomenon, the answer primarily depends on how the product documentation enables co-development and replication. This article seeks to provide guidance on which information sufficiently describes OSH. In other words, what is the source of OSH?\n<\/p><p>The objective of this article is to provide an overview of how current projects tend to interpret and make use of the concept of OSH. Its ultimate goal is to provide a deeper description of what OSH means based on the observation of actual practices. This is performed through the analysis of the \u201csource\u201d, i.e., the published documentation of 132 OSH products with the help of categorical criteria addressing the question \u201chow open are OSH products?\u201d It specifically focuses on the recent evolution of the open-source movement outside the domain of electronics and DIY to those of non-electronic and complex OSH products.\n<\/p><p>The remainder of the paper is structured as follows. In the next section the general context of emergence of OSH as an alternative product development pattern based on free distribution of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> is depicted and characterized. Then definitions from practice communities and scholars are analysed and combined in order to provide a consolidated overview of the concept of OSH. After the definitions, the methodological approach for the acquisition of empirical data allowing the analysis of current practices of OSH documentation is introduced. The results produced by the application of this method are then described and interpreted. Finally, we summarize the findings into an original framework termed \"OSH lifecycle,\" additionally summarizing observed approaches to OSH.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_context_of_open-source_hardware\">The context of open-source hardware<\/span><\/h2>\n<p>OSH is a relatively young phenomenon with projects emerging in the past decade<sup id=\"rdp-ebb-cite_ref-BalkaOpen16_8-0\" class=\"reference\"><a href=\"#cite_note-BalkaOpen16-8\">[7]<\/a><\/sup>, although it has several prominent examples already. Pioneering projects such as RepRap, Open Source Ecology, and Local Motors have certainly set a precedence to lift the air of mystery and aloofness of engineering ingenuity closely guarded for means of commercial appropriation. As to whether these are heralds of what Moritz <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MoritzOpen15_9-0\" class=\"reference\"><a href=\"#cite_note-MoritzOpen15-9\">[8]<\/a><\/sup> depict as disruptive changes on the upstream end of value chains toward value-co-creation, only time will tell. Clearly, this alternative course of action could take an active part in shaping the technological future. Indeed, there are already promising examples of successful businesses based on OSH for which the scientific community identified corresponding value creation models.<sup id=\"rdp-ebb-cite_ref-PearceEmerging17_10-0\" class=\"reference\"><a href=\"#cite_note-PearceEmerging17-10\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiWhyOpen17_11-0\" class=\"reference\"><a href=\"#cite_note-LiWhyOpen17-11\">[10]<\/a><\/sup> These examples, together with the empirical evidence provided by free and open-source software, allow for a foreseeing of a flourishing future for OSH.<sup id=\"rdp-ebb-cite_ref-PearceEmerging17_10-1\" class=\"reference\"><a href=\"#cite_note-PearceEmerging17-10\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiWhyOpen17_11-1\" class=\"reference\"><a href=\"#cite_note-LiWhyOpen17-11\">[10]<\/a><\/sup>\n<\/p><p>Like free and open-source software, OSH is an IT-enabled internet phenomenon. Fjeldsted <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-FjeldstedOpen12_12-0\" class=\"reference\"><a href=\"#cite_note-FjeldstedOpen12-12\">[11]<\/a><\/sup>, as well as Bonvoisin and Boujut<sup id=\"rdp-ebb-cite_ref-BonvoisinOpen15_13-0\" class=\"reference\"><a href=\"#cite_note-BonvoisinOpen15-13\">[12]<\/a><\/sup>, point out the integral part IT platforms play in fostering product-related data sharing and community-based product development, as well as the emergence of OSH-based business models. However, Raasch<sup id=\"rdp-ebb-cite_ref-RaaschProduct11_4-1\" class=\"reference\"><a href=\"#cite_note-RaaschProduct11-4\">[4]<\/a><\/sup> draws a contrast: compared with OSS development, the aspect of physical object design in OSH has strong impacts on required skills, tools, and infrastructure. This aspect is even increasingly salient as the focus of OSH progressively expands towards many other forms of hardware than electronic hardware, like mechanical, construction, medical, optical, agricultural, or textile hardware. From a design point-of-view, the degree of freedom in the problem-solution space introduced by their mechanical portion rises significantly. Howard <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-HowardOpen12_14-0\" class=\"reference\"><a href=\"#cite_note-HowardOpen12-14\">[13]<\/a><\/sup> and Raasch, Herstatt, and Balka<sup id=\"rdp-ebb-cite_ref-RaaschOnThe09_15-0\" class=\"reference\"><a href=\"#cite_note-RaaschOnThe09-15\">[14]<\/a><\/sup> also illustrate how OSH is looking at the full spectrum of product complexity and different manufacturing strategies, from do-it-yourself (DIY) to industrial production.\n<\/p><p>The capacity of communities outside the closed and hierarchical environment of research and development (R&D) departments to develop complex and high-quality products remains a challenge; the largest majority of OSH products available to date are gadgets for hobbyists.<sup id=\"rdp-ebb-cite_ref-HansenTheCurrent13_16-0\" class=\"reference\"><a href=\"#cite_note-HansenTheCurrent13-16\">[15]<\/a><\/sup> As processes transform inputs into outputs, their structure is generally related to the product and project scope. OSH can be described as individual or participatory realization of a shared product design. Therefore, 3D printing designs shared by individuals on sites such as Thingiverse<sup id=\"rdp-ebb-cite_ref-17\" class=\"reference\"><a href=\"#cite_note-17\">[b]<\/a><\/sup> may classify as OSH, similar to large community-based efforts such as the E-Nable project.<sup id=\"rdp-ebb-cite_ref-18\" class=\"reference\"><a href=\"#cite_note-18\">[c]<\/a><\/sup> This makes a big difference in regards to the type of effort being structured for the realization of an open-source product. Hence, depending on the product scope and also the originator's intent to foster co-design, levels of interaction maturity differ starkly in OSH. A useful macro-level classification is offered by Camarinha-Matos <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Camarinha-MatosCollab09_19-0\" class=\"reference\"><a href=\"#cite_note-Camarinha-MatosCollab09-19\">[16]<\/a><\/sup>, starting at networks, then moving to coordination, cooperation, and collaboration.\n<\/p><p>The conceptual space that supports collective spirit and collaborative problem solving is described by Maher, Paulini, and Murty.<sup id=\"rdp-ebb-cite_ref-MaherScaling10_20-0\" class=\"reference\"><a href=\"#cite_note-MaherScaling10-20\">[17]<\/a><\/sup> It is based on the three aspects of shared representation, motivation, and communication. They enable problem solving in collective design projects within a continuum from collected (individual) to collective intelligence. The latter is demarcated by collaborative generation of solutions as well as synthesis of individual solutions. From their study of collective design activities<sup id=\"rdp-ebb-cite_ref-PauliniDesign13_21-0\" class=\"reference\"><a href=\"#cite_note-PauliniDesign13-21\">[18]<\/a><\/sup> conclude that within the frame of what they call an inclusive nature of participatory design, individuals proactively self-organize and choose their own roles as well as period of involvement. The maturity of this participatory nature may in fact be the primary determinant of collective design projects.\n<\/p><p>Social dynamics in crowds are context-dependent, however. In the OSH context, from all of the above, communities can be viewed as socially formed groups of heterogeneous actors who co-create OSH products. Depending on the participatory roles of actors, they may engage as followers, replicators, developers, or community managers. In the frame of open innovation, West and Lakhani<sup id=\"rdp-ebb-cite_ref-WestGetting08_22-0\" class=\"reference\"><a href=\"#cite_note-WestGetting08-22\">[19]<\/a><\/sup> describe them as actor volunteers lacking of organizational affiliation. Due to partial influence of market-based factors, this role may somewhat reflect the notion of people giving away their time and effort without any monetary compensation. In addition, actors may as well include individuals, firms, and many other types of organizations.<sup id=\"rdp-ebb-cite_ref-vonHippelDemoc06_23-0\" class=\"reference\"><a href=\"#cite_note-vonHippelDemoc06-23\">[20]<\/a><\/sup>\n<\/p><p>Aksulu and Wade<sup id=\"rdp-ebb-cite_ref-AksuluAComp10_24-0\" class=\"reference\"><a href=\"#cite_note-AksuluAComp10-24\">[21]<\/a><\/sup> provide insight on how pure open-source systems set a community-based context of personal development, process learning, and effective technology outputs. In contrast, within the purely proprietary context, purpose is realized by the efficient generation of technology outputs within organizations with clear boundaries.\n<\/p><p>What exactly makes open-source processes effective seems to be rooted in the task environment. According to Lee and Cole<sup id=\"rdp-ebb-cite_ref-LeeFromAFirm03_25-0\" class=\"reference\"><a href=\"#cite_note-LeeFromAFirm03-25\">[22]<\/a><\/sup>, the trade-off between exploration and exploitation is made through a two-tier task structure of generated variations within the periphery, and selection and retention at the core. In their case study of the Linux kernel development project, they observe community-based knowledge creation in the form of an evolutionary learning process based on a culture of critical evaluations and peer learning. Moreover, Howison and Crowston<sup id=\"rdp-ebb-cite_ref-HowisonCollab14_26-0\" class=\"reference\"><a href=\"#cite_note-HowisonCollab14-26\">[23]<\/a><\/sup> describe collaborative development of OSS as an evolutionary process of incremental and independent tasks, by which a layer structure emerges task-by-task. They argue that added layers change the context over time and significantly reduce complexity of tasks. By gradually laying the foundation, previously unattainable tasks eventually may suddenly become easily achievable by individual developers, or simple workarounds may become obvious. This is in line with the concept of Maher <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MaherForm96_27-0\" class=\"reference\"><a href=\"#cite_note-MaherForm96-27\">[24]<\/a><\/sup> concerning the co-evolution of the problem and solution spaces, which describes how these dimensions influence each other during the design process. Collaborative evolutionary design on the activity level is key in explaining how solutions are generated effectively in open-source systems<sup id=\"rdp-ebb-cite_ref-28\" class=\"reference\"><a href=\"#cite_note-28\">[d]<\/a><\/sup> and needs to be further researched.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Definitions_of_openness_from_practice_and_research\">Definitions of openness from practice and research<\/span><\/h2>\n<p>This section reviews existing definitions from practice and identifies resulting implications in terms of published product-related documentation. These definitions are then matched with existing contributions from research in order to provide a comprehensive framework defining OSH. Results of empirical studies which challenge these definitions and show a more heterogeneous landscape of OSH are then presented and lead to the identification of two research questions addressed in this article.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Definitions_from_practice_communities\">Definitions from practice communities<\/span><\/h3>\n<p>The Open Source Hardware Statement of Principles 1.0 states that \u201copen source hardware is hardware whose design is made publicly available so that anyone can study, modify, distribute, make, and sell the design or hardware based on that design.\u201d<sup id=\"rdp-ebb-cite_ref-OSHADefinition16_29-0\" class=\"reference\"><a href=\"#cite_note-OSHADefinition16-29\">[25]<\/a><\/sup> It is based on the assumption that publishing a \u201cdesign\u201d (alternatively termed \u201cdocumentation\u201d) realizes the four freedoms of the open-source concept, which are reinterpreted in the context of tangible products as the freedom to study, modify, make, and distribute.<sup id=\"rdp-ebb-cite_ref-31\" class=\"reference\"><a href=\"#cite_note-31\">[e]<\/a><\/sup>\n<\/p><p>This definition is an adaptation of the original Open Source Definition by the Open Source Initiative<sup id=\"rdp-ebb-cite_ref-OSITheOpen07_32-0\" class=\"reference\"><a href=\"#cite_note-OSITheOpen07-32\">[27]<\/a><\/sup> to the realms of tangible products. Coined in the context of free and open-source software, it specifies requirements on two interwoven objects: the \u201csoftware,\u201d alternatively called the \u201cproduct,\u201d and the \u201csource code.\u201d It is implicitly accepted that the object called \u201csource code\u201d allows defining the object called \u201cproduct\u201d unambiguously in its depth and its entirety. In the case of software, this implicit prerequisite is automatically fulfilled, as a software product is the translation of a text written in programming language into a machine language by a deterministic algorithm. However, this is not the case for tangible products: what information has to be shared in order to allow any interested person to study, modify, make, and distribute a piece of hardware is not a simple question.\n<\/p><p>A closer look at the definitions of widely recognized OSH licences and current practices of OSH seeks to give some hints and shows that the four freedoms of open source tend to be supported by different document types or properties<sup id=\"rdp-ebb-cite_ref-BonvoisinBestPrac17_33-0\" class=\"reference\"><a href=\"#cite_note-BonvoisinBestPrac17-33\">[28]<\/a><\/sup>:\n<\/p>\n<ul><li> The freedom to study (i.e., the right to access sufficient information to understand how the piece of hardware\u2014referred herein as the product\u2014works and to retrace the underlying design rationale as defined by Wang, Johnson, and Bracewell<sup id=\"rdp-ebb-cite_ref-WangTheRet12_34-0\" class=\"reference\"><a href=\"#cite_note-WangTheRet12-34\">[29]<\/a><\/sup>) can be supported by the publication of schematics, 2D, or 3D CAD files.<\/li>\n<li> The freedom to modify (i.e., the right to edit the product definition documents and to tweak or develop the product further for any purpose) can be supported by the publication of all documents in their original editable format.<\/li>\n<li> The freedom to make (i.e., the right to use the product definition documents in order to make\u2014in other words to produce, to manufacture\u2014the piece of hardware) can be supported by the publication of a bill of materials and assembly instructions.<\/li>\n<li> The freedom to distribute (i.e., the right to give or sell the product definition documents as well as the physical products fabricated with the help of these documents) is allowed by the publication of all documents under a licence which grants free redistribution, including for commercial purposes.<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Definitions_from_scholarly_research\">Definitions from scholarly research<\/span><\/h3>\n<p>Scholarly definitions\u2014mostly stemming from innovation management research\u2014deliver categories which are consistent with the Open Source Hardware Statement of Principles 1.0 without, however, giving further details on the nature of the documentation to be published.\n<\/p><p>Freedoms to study, to edit, and to make are consistent with the three factors of openness defined by Balka, Raasch, and Herstatt.<sup id=\"rdp-ebb-cite_ref-BalkaTheEffect14_5-1\" class=\"reference\"><a href=\"#cite_note-BalkaTheEffect14-5\">[5]<\/a><\/sup> They respectively term these factors \"transparency,\" \"accessibility,\" and \"replicability.\" Transparency refers to the possibility for any interested person to access sufficient information to understand the product in detail without restriction (which equals the freedom to study). Accessibility refers to the possibility for any interested person to edit design information and therefore to further develop the product (which equals the freedom to modify). Replicability refers to the possibility for any interested person to physically produce the product (which equals the freedom to make). These three factors of openness are introduced as preconditions for three complementary behaviors on the part of the community surrounding the product: Transparency enables observation (and eventually feedback), accessibility enables further development (and eventually co-development), and replicability enables prototyping and production (and eventually co-production). What is missing in this contribution is the fourth factor of openness termed henceforth \"commercial usability,\" which addresses the above listed freedom to distribute.\n<\/p><p>The four freedoms and corresponding factors of openness cover both \u201cdistinct and competing meanings\u201d of openness identified by von Hippel<sup id=\"rdp-ebb-cite_ref-vonHippelComment10_35-0\" class=\"reference\"><a href=\"#cite_note-vonHippelComment10-35\">[30]<\/a><\/sup>: the permeability of the innovation process to the participation of external people and the public sharing of documentation. Transparency, replicability, and commercial usability are about sharing documentation publicly. Accessibility is about enabling the participation of external people in the design process. Aitamurto, Holland, and Hussain<sup id=\"rdp-ebb-cite_ref-AutamurtoTheOpen15_36-0\" class=\"reference\"><a href=\"#cite_note-AutamurtoTheOpen15-36\">[31]<\/a><\/sup> term two meanings of openness: process openness (whether the innovation process is open or closed) and product openness (whether the innovation outcome is open or closed). Huizingh<sup id=\"rdp-ebb-cite_ref-HuizinghOpen11_37-0\" class=\"reference\"><a href=\"#cite_note-HuizinghOpen11-37\">[32]<\/a><\/sup> defines open-source innovation as a result of both process and product openness. Raasch, Herstatt, and Balka<sup id=\"rdp-ebb-cite_ref-RaaschOnThe09_15-1\" class=\"reference\"><a href=\"#cite_note-RaaschOnThe09-15\">[14]<\/a><\/sup> particularly deliver the following definition: \u201copen-source innovation is characterized by free revealing of information on a new design with the intention of collaborative development of a single design or a limited number of related designs for market or nonmarket exploitation.\u201d\n<\/p><p>The coherent picture of OSH drawn by those definitions from practice and scholarly research is displayed in Figure 1. OSH builds upon product and process openness. Product openness results from the factors transparency, replicability, and commercial usability, which respectively address the freedoms to study, make, and distribute. Process openness results from the sole factor accessibility, which addresses the freedom to edit.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Bonvoisin_JOfOpenHard2017_1-1.png\" class=\"image wiki-link\" data-key=\"a4f63452abc9b2add109bee61f817d1e\"><img alt=\"Fig1 Bonvoisin JOfOpenHard2017 1-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/50\/Fig1_Bonvoisin_JOfOpenHard2017_1-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Forms of openness involved in open-source hardware<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Theory-practice-gap\">Theory-practice-gap<\/span><\/h3>\n<p>The few existing empirical studies of OSH deliver a picture from practice observations that is less strict than the ambitious theoretical approach summarized by Figure 1, beyond the misuses, which can be expected with the upcoming of a new concept.\n<\/p><p>Balka, Raasch, and Herstatt<sup id=\"rdp-ebb-cite_ref-BalkaTheEffect14_5-2\" class=\"reference\"><a href=\"#cite_note-BalkaTheEffect14-5\">[5]<\/a><\/sup> showed that factors of openness may be valued differently by development community members in the fields of consumer electronics and IT hardware. In the communities they analyzed, accessibility of software tended to play the largest role in the involvement of members, while transparency and replicability of software played a secondary role, and the openness of the related hardware played no role. They also highlighted that, as a result of the low importance given to hardware in those communities, hardware components were generally \u201cless documented\u201d than the software components. This raises the following research question:\n<\/p>\n<blockquote>RQ1 \u2013 To what extent are the four aspects of openness reflected by practitioners in the publication of product-related documentation? Do practitioners tend to employ the whole set of these factors, or do they tend to employ only subsets of these factors?<\/blockquote>\n<p>Based on interviews with originators of OSH products, Bonvoisin <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BonvoisinCurrent17_38-0\" class=\"reference\"><a href=\"#cite_note-BonvoisinCurrent17-38\">[33]<\/a><\/sup> defined the contours of two archetypes of projects developing OSH they termed \"development communities\" and \"isolated innovators.\" Development communities are characterized by high process openness, adopt systematic and early release policies, and gather contributions from a larger and permeable group of people. On the contrary, isolated innovators are characterized by low levels of process openness. They do not gather contributions from the outside, but rather they release complete product documentations after product versions are fully developed. While the former are interested in integrating people from the outside during the product development process, the latter are more interested in broadcasting their innovation, i.e., enabling other people to produce it. This contradicts the statement made by Gacek and Arief<sup id=\"rdp-ebb-cite_ref-GacekTheMany04_39-0\" class=\"reference\"><a href=\"#cite_note-GacekTheMany04-39\">[34]<\/a><\/sup> that \u201cthe term 'open source' has been widely used to describe a software development process.\u201d Findings reported by Bonvoisin <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BonvoisinCurrent17_38-1\" class=\"reference\"><a href=\"#cite_note-BonvoisinCurrent17-38\">[33]<\/a><\/sup> suggest on the contrary that motivations for publishing documentation may depend on the phase of the product lifecycle in which the community is supposed to play a role and raise the following second research question:\n<\/p>\n<blockquote>RQ2 \u2013 If factors of openness are understood by practitioners as optional, can we find typical patterns in the way practitioners decide to comply with specific factors of openness? Are these aspects related to specific product lifecycle phases?<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Research_approach_and_methodology\">Research approach and methodology<\/span><\/h2>\n<p>In order to address the research questions defined above, an empirical approach based on the systematic analysis of a representative group of OSH products, as well as their published documentation, has been adopted. This approach is similar to those adopted by Balka, Raasch, and Herstatt<sup id=\"rdp-ebb-cite_ref-BalkaTheEffect14_5-3\" class=\"reference\"><a href=\"#cite_note-BalkaTheEffect14-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BalkaOpen09_40-0\" class=\"reference\"><a href=\"#cite_note-BalkaOpen09-40\">[35]<\/a><\/sup> while avoiding two pitfalls limiting the validity of the produced results. The first is the use of unclear categories such as the size of the development community. As shown in Bonvoisin <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BonvoisinCurrent17_38-2\" class=\"reference\"><a href=\"#cite_note-BonvoisinCurrent17-38\">[33]<\/a><\/sup>, belonging to a development community is a fickle concept which may be based on different types of activities (from pressing the button \u201cI like\u201d to bringing significant contribution to the product design), which are highly flexible over time and therefore difficult to measure objectively.<sup id=\"rdp-ebb-cite_ref-41\" class=\"reference\"><a href=\"#cite_note-41\">[f]<\/a><\/sup> The second is the use of subjective evaluation of the openness factors with the help of Likert scales. In contrast to the approaches of Balka, Raasch, and Herstatt<sup id=\"rdp-ebb-cite_ref-BalkaTheEffect14_5-4\" class=\"reference\"><a href=\"#cite_note-BalkaTheEffect14-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BalkaOpen09_40-1\" class=\"reference\"><a href=\"#cite_note-BalkaOpen09-40\">[35]<\/a><\/sup>, the research reported here strives at a systematic analysis of published documentation with the help of measurable indicators, from the part of an impartial observer.\n<\/p><p>In the following subsections, the approach adopted for the acquisition and the analysis of empirical data is presented. The first subsection introduces methodological choice made in focusing on non-electronic complex hardware products. The second subsection describes the method applied for the establishment of a representative pool of those products. The third subsection defines a set of criteria used to characterize products and their related documentation. The fourth subsection defines criteria for the assessment of openness factors based on the acquired data. The last subsection introduces a clustering method used for identifying typical patterns in the publication of OSH documentation.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Focus:_Complex_non-electronic_hardware_products\">Focus: Complex non-electronic hardware products<\/span><\/h3>\n<p>While the transfer of open-source principles from software to hardware historically and logically started with electronic hardware<sup id=\"rdp-ebb-cite_ref-GibbBuild14_42-0\" class=\"reference\"><a href=\"#cite_note-GibbBuild14-42\">[36]<\/a><\/sup>, other technologies are increasingly impacted by the phenomenon. The extension of open-source practices to non-electronic hardware such as mechanical products is of particular interest in terms of documentation. Electronic hardware is a thoroughly standardized field where components are purchased off the shelf. But this is not the case for mechanical hardware, where non-standardized free form components play a large role.\n<\/p><p>Also, particularly interesting is the consideration of complex products. Product complexity is defined by Jacobs<sup id=\"rdp-ebb-cite_ref-JacobsProduct07_43-0\" class=\"reference\"><a href=\"#cite_note-JacobsProduct07-43\">[37]<\/a><\/sup> as \u201ca design state resulting from the multiplicity of, and relatedness among, product architectural elements.\u201d This characteristic influences the level of professionalization required in the development and production of a given product. It also determines whether production can take place in either DIY or industrial production settings. Noteworthy is that, while DIY and OSH are two interwoven phenomena, not every OSH product is meant to be produced in a DIY production setting.<sup id=\"rdp-ebb-cite_ref-BonvoisinDesign17_44-0\" class=\"reference\"><a href=\"#cite_note-BonvoisinDesign17-44\">[38]<\/a><\/sup> Product complexity also relates to design effort in terms of resources consumed and process duration.<sup id=\"rdp-ebb-cite_ref-Rodriguez-ToroProduct04_45-0\" class=\"reference\"><a href=\"#cite_note-Rodriguez-ToroProduct04-45\">[39]<\/a><\/sup> Highly complex products tend to require inputs from multiple people and are more relevant for the topic of collaborative design.\n<\/p><p>In order to reflect the challenges faced by OSH products in terms of documentation, the methodological approach pursued in this article is to focus on the most challenging range of products. The underlying assumption is that \u201cwho can do more can do less,\u201d so what applies to complex non-electronic hardware applies also to more electronic hardware and simple products. As a consequence, this article specifically focuses on the recent evolution of the open-source movement outside the domain of electronics and DIY to those of non-electronic and complex OSH products. It excludes the field of purely electronic hardware products. Not only is this methodological choice in consistency with the academic background of the authors situated in mechanical engineering and collaborative engineering design; it is also in line with the most recent advancement of the open-source concept on forms of hardware which are in the sphere of influence in these disciplines. It should be borne in mind that the huge achievements of the still much larger open electronic hardware field have paved the way for this development. In no way is the exclusion of this field in this study an indicator of its irrelevance. Rather, it is a matter of limiting the extent of the research undertaking.\n<\/p><p>As a result of this position, in the rest of the paper, \u201copen source hardware\u201d is used to refer to complex, non-electronic OSH.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Product_selection\">Product selection<\/span><\/h3>\n<p>In contrast to previous research works aiming at grasping the contours of a phenomenon related to open source, e.g., sharing of 3D-printed designs (see Kyriakou, Nickerson, and Sabnis<sup id=\"rdp-ebb-cite_ref-KyriakouKnowledge17_46-0\" class=\"reference\"><a href=\"#cite_note-KyriakouKnowledge17-46\">[40]<\/a><\/sup>), no major centralized online entry point exists which aggregates relevant products in the context of OSH. While platforms such as GitHub or Thingiverse are playing this role for free and open-source software or 3D-printing communities, to date, there is no generally acknowledged and widely used online platform supporting open-source product development. These observations have been respectively made by Raasch<sup id=\"rdp-ebb-cite_ref-RaaschProduct11_4-2\" class=\"reference\"><a href=\"#cite_note-RaaschProduct11-4\">[4]<\/a><\/sup> and Howard <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-HowardOpen12_14-1\" class=\"reference\"><a href=\"#cite_note-HowardOpen12-14\">[13]<\/a><\/sup> and are still valid. On the contrary, projects developing OSH products tend to use a wide variety of tools for collaboration and dissemination.<sup id=\"rdp-ebb-cite_ref-BonvoisinCurrent17_38-3\" class=\"reference\"><a href=\"#cite_note-BonvoisinCurrent17-38\">[33]<\/a><\/sup>\n<\/p><p>As a result, OSH products have been identified using conventional internet search engines using a snowball-effect approach. The results of this search have been screened through a restrictive criteria set in order to ensure a conservative evaluation of the phenomenon to be studied and to keep a deliberately narrow focus within the highly diverse field of open-source innovation (see for example Ehls<sup id=\"rdp-ebb-cite_ref-EhlsDiversity15_47-0\" class=\"reference\"><a href=\"#cite_note-EhlsDiversity15-47\">[41]<\/a><\/sup> on that topic). The following selection criteria have been applied:\n<\/p>\n<ul><li> The product is the outcome of discrete manufacturing. Products of food and process industries such as yogurt, cement, chemicals, or plastic compounds are excluded.<\/li><\/ul>\n<ul><li> The product contains at least in-house, tangible, and non-electronic hardware, which includes mechanical or any other type of non-electronic physical elements (e.g., textile). It may eventually include electronic hardware and consequently software. Purely electronic hardware or software products such as Arduino or Linux are therefore excluded. In-house means that the piece of non-electronic hardware is \"original equipment\" created by the product originator and not a component bought off the shelf.<\/li><\/ul>\n<ul><li> The product is complex. While well-defined metrics assessing product complexity on a positive real scale have been proposed in the scientific literature (e.g., Rodriguez-Toro, Jared, and Swift <sup id=\"rdp-ebb-cite_ref-Rodriguez-ToroProduct04_45-1\" class=\"reference\"><a href=\"#cite_note-Rodriguez-ToroProduct04-45\">[39]<\/a><\/sup>), using those metrics requires having sufficient access to detailed data such as the number of parts. This condition is difficult to satisfy in the context of observation of partly documented and defined products. Therefore, in the following analysis, the complexity of products is evaluated with a rule of thumb: products are considered as complex if they consist of more than two parts of different materials. Products such as business card holders, cell phone cases, or other 3D-printed gimmicks are out of scope. The objective here is to bring to the foreground those open-source products that are on the upper side of the complexity scale. In other words, to select the few complex ones and let aside the myriad of gimmicks that can be found on CAD file exchange platforms such as Thingiverse or Shapeways.<\/li><\/ul>\n<ul><li> The product is developed for functional rather than aesthetic purposes. Jewelry, artistic, and decorative items do not fulfill this criterion and therefore were not included.<\/li><\/ul>\n<ul><li> The product is at least partly defined, i.e., is provided with sufficient documentation to indicate that the product development maturity is equivalent or above the stage \u201csystem-level design\u201d of the product development process as defined by Ulrich and Eppinger.<sup id=\"rdp-ebb-cite_ref-UlrichProduct11_48-0\" class=\"reference\"><a href=\"#cite_note-UlrichProduct11-48\">[42]<\/a><\/sup> Undeveloped product concepts are not considered. Thus, the focus delineates them from challenge platforms, which focus solely on capturing concepts from external participants.<\/li><\/ul>\n<ul><li> The product is labelled by its surrounding community as open source. The terms \u201copen source,\u201d \u201copen-source hardware,\u201d or \u201copen hardware\u201d are used in the published documentation as an adjective to qualify either the product or the activities around the product. This is quite a conservative criterion. A non-negligible number of projects adopt principles of OSH without necessarily fitting neatly into this conservative criterion.<\/li><\/ul>\n<ul><li> There are no systematic 1:1 relations between product, project, and community. For example, a community may be involved in more than one project and a project may involve the development of more than one product. However, in order to simplify the analysis, only one product per project and per community has been considered.<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Data_acquisition\">Data acquisition<\/span><\/h3>\n<p>For each of the selected products, two sets of descriptive criteria are defined. These are summarized in Table 1. The first set is dedicated to the characterization of the published product-related information. It translates in measurable terms the best practices of OSH as provided by the Open Source Hardware Association<sup id=\"rdp-ebb-cite_ref-OSHABest13_49-0\" class=\"reference\"><a href=\"#cite_note-OSHABest13-49\">[43]<\/a><\/sup> and complemented by a practice review performed prior to the reported research.<sup id=\"rdp-ebb-cite_ref-BonvoisinBestPrac17_33-1\" class=\"reference\"><a href=\"#cite_note-BonvoisinBestPrac17-33\">[28]<\/a><\/sup> In the absence of practicable indicators for assessing the quantity, the quality, or the relevance of a piece of information, simple binary criteria are used, which indicate the presence or the absence of a given piece of information or of a given property. The second set delivers contextual information about the product, the corresponding development project, as well as the surrounding product development community.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 1.<\/b> Open-source hardware product characterization criteria.\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Ref.\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Name\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Description\n<\/th><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"3\">Part I \u2013 Criteria regarding shared product-related documentation\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>a<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CAD files available\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether CAD files or schematics of the non-electronic hardware are available online\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>b<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CAD files aeditable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether the online released CAD files of the product are editable. CAD files are considered editable if they are released in their original format. They are not considered editable if they are only released in an export format such as PDF or STL, which does not allow further modifications.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>c<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assembly instructions available\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether instructions for building the non-electronic hardware are available online\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>d<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assembly instructions editable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether the published assembly instructions are editable. Assembly instructions are considered editable if they can be edited in a web 2.0 environment or downloaded as editable files. A file is furthermore considered editable if it is released in its original format. It is not considered as editable if it is only available in an export format such as PDF.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>e<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bill of materials available\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether a bill of materials relative to the non-electronic hardware is available online\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>f<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bill of materials editable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether the published bill of materials is editable. A bill of materials is considered editable if it can be edited in a web. 2.0 environment or downloaded as an editable file. A file is furthermore considered editable if it is released in the original format. It is not considered editable if it is only available in an export format such as PDF.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>g<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Guidelines for participation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether guidelines for participation or a dedicated call for contribution are provided to potential contributors\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>h<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Commercial usage allowed\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether the licence applied to the non-electronic hardware allows commercial usage of the published content. If no licence is applied, the criterion is set to false.\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"3\">Part II \u2013 Criteria regarding contextual information\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>i<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">License\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Licensing scheme used for the publication of the non-electronic hardware\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>j<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Contains electronics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Boolean value indicating whether the product contains electronic hardware components as well\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>k<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Maturity\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Defines the maximum maturity level achieved by one of the eventual versions of the product over time. Five maturity levels are defined in the order of increasing liability risk:<br \/> <br \/>\n<p>1. Design - There is only a theoretical design that is still to be fully developed. No liability of the product originator applies.<br \/>\n2. Prototype - The early design phases have been completed and the first functional prototype has been built. No liability of the product originator applies.<br \/>\n3. Production\/DIY - The product is fully defined and documented and can be replicated. No liability of the product originator applies.<br \/>\n4. Production\/Kit - The product is sold by a commercial actor as a kit. Limited availability applies to the vendor.<br \/>\n5. Production\/full product - The product is sold by a company as a finished product. Full availability applies to the vendor.\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>l<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Status of the community\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Binary value indicating whether the community is active. The community is considered inactive in case no activity (encompassing either product development or sales and marketing) can be detected on the website or the collaboration platforms within one year.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>m<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Product category\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classification of the products in product groups\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Criteria are evaluated based on information that can be accessed through the websites of the product development communities. For the four criteria \u201cCAD files available,\u201d \u201cassembly instructions available,\u201d \u201cbill of materials available,\u201d and \u201cguidelines for participation,\u201d the following practical cut-off rule is applied for each binary criterion: If the necessary information to satisfy a criterion cannot be found in less than 10 minutes, it is considered unavailable. This cut-off rule is consistent with the fact that accessibility to documents required by the Open Source Definition implies not only that these documents can be accessed, but also that they can be found easily.\n<\/p><p>Note that with the focus of the prior subsection titled \u201cFocus: Complex non-electronic hardware products,\u201d solely the documentation relative to the non-electronic hardware has been assessed. Documentation regarding electronic hardware and software has not been regarded. The authors emphasize here that this does not mean that the openness of electronic hardware has lower relevance in general. This methodological choice is instead dictated by the background of the authors and the willingness to limit volume of data to be gathered in this study. This attempt is based on the assumption that data gathered for non-electronic hardware are representative for data about electronic hardware.\n<\/p><p>Also, this article specifically focuses on OSH development. The use of OSH may require pieces of documentation which are not considered here, such as operating instructions and parts specifications. Withholding this information may render a product useless or even unsafe. The provision of operating instructions is for this reason legally requested in specific product branches such as an automobile. As a consequence, the provision of operational instructions has not been considered as a distinctive characteristic of OSH in this article.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Openness_assessment_criteria\">Openness assessment criteria<\/span><\/h3>\n<p>Based on the data defined in previous sections, four composite criteria are defined for assessing the compliance of the product-related documentation to the four factors of openness:\n<\/p>\n<ul><li> The transparency (<i>T<\/i>) criterion addresses the freedom to study. A product satisfies this criterion when CAD files are published.<\/li>\n<li> The accessibility (<i>A<\/i>) criterion addresses the freedom to modify. A product satisfies this criterion when all published content is editable or when a guideline for participation is available.<\/li>\n<li> The replicability (<i>R<\/i>) criterion addresses the freedom to make. A product satisfies this criterion when assembly instructions and bill of materials are available.<\/li>\n<li> The commercial usability (<i>C<\/i>) criterion addresses the freedom to distribute. A product satisfies this criterion when licences applied to the non-electronic hardware allow commercial usage of the published content.<\/li><\/ul>\n<p>The four criteria are defined as logical operations on the Boolean values [<i>a<\/i>,\u2026,<i>h<\/i>] as depicted by the following equation:\n<\/p><p><br \/>\n<span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/521705eae2abf8feb44a54cc85a1ab83d9bef0f3'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -5.838ex; width:54.292ex; height:12.843ex;\" \/><\/span>\n<\/p><p><br \/>\nIn addition to this, a global openness index (<i>OI<\/i>) is defined as a cumulative point system. A product gets one point each time one of the criteria <i>a<\/i>, <i>b<\/i>, <i>c<\/i>, <i>d<\/i>, <i>e<\/i>, <i>f<\/i>, <i>g<\/i>, and <i>h<\/i> is satisfied. An OI = 8 means the product fully satisfies the best practices of OSH. An OI = 0 means the product satisfies none of the criteria defining OSH. The openness index is defined in the next equation:\n<\/p><p><br \/>\n<span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/401dbc8b49881259c8ba76956430cc2de726df0c'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -5.838ex; width:43.613ex; height:12.843ex;\" \/><\/span>\n<\/p><p><br \/>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Clustering\">Clustering<\/span><\/h3>\n<p>In order to identify typical patterns in the publication of product-related documentation, products are clustered according to the values of the eight binary criteria [<i>a<\/i>,\u2026,<i>h<\/i>] defined above. The k-medioids algorithm (or PAM, Partitioning Around Medioids) has been used since it is particularly adapted to the clustering of objects described by categorical data. This method requires as input an <i>n<\/i>*<i>n<\/i> matrix giving a measure of the similarity of each pair of product. Each product is represented by a binary word of eight bits, with each bit representing a criterion from <i>a<\/i> to <i>h<\/i>, as described in the following equation:\n<\/p><p><br \/>\n<span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/0c5beb8deb59439537290e5da518f321a78fe2f1'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -0.671ex; width:14.737ex; height:2.509ex;\" \/><\/span>\n<\/p><p>Where:\n<\/p>\n<ul><li> <i>P<sub>i<\/sub><\/i> is an eight-bit word representing the product <i>i<\/i>; and<\/li>\n<li> <i>a<sub>i<\/sub><\/i>, \u2026, <i>h<sub>i<\/sub><\/i> are the binary values of criteria <i>a<\/i> to <i>h<\/i> evaluated for product <i>i<\/i>.<\/li><\/ul>\n<p><br \/>\nInter-product similarity is computed according to the Manhattan distance<sup id=\"rdp-ebb-cite_ref-ChoiASurv10_50-0\" class=\"reference\"><a href=\"#cite_note-ChoiASurv10-50\">[44]<\/a><\/sup> of the products\u2019 descriptive 8-bit words as described in the following equation:\n<\/p><p><br \/>\n<span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/15b53923408173f7646e34b7aa46a17e1e513bab'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -3.005ex; width:42.502ex; height:7.343ex;\" \/><\/span>\n<\/p><p>Where:\n<\/p>\n<ul><li> <i>S<sub>ij<\/sub><\/i> is the Manhattan distance of binary words <i>P<sub>i<\/sub><\/i> and <i>P<sub>j<\/sub><\/i> and is a numerical of [0, 8]. A distance of 0 means words are equal. A distance of 8 means words are completely different.<\/li>\n<li> <i>Pi,k<\/i> and <i>Pj,k<\/i> are respectively the <i>k<\/i><sup>th<\/sup> bit of words <i>P<sub>i<\/sub><\/i> and <i>P<sub>j<\/sub><\/i>; and<\/li>\n<li> <i>\u0393<\/i> is the function defined in the second equation.<\/li><\/ul>\n<p><br \/>\nThe k-medioids algorithm requires also as parameter the number of clusters to be defined. The optimal number of clusters\u2014so that adding another cluster does not provide further information\u2014is calculated with the help of the elbow method.<sup id=\"rdp-ebb-cite_ref-TibshiraniEstim02_51-0\" class=\"reference\"><a href=\"#cite_note-TibshiraniEstim02-51\">[45]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>A data acquisition campaign was carried out between March 2016 and April 2017. This large time period ensured sufficient exhaustiveness of the snow-ball research. At the same time, it generated risks of data obsolescence. Therefore, the gathered data set was screened and actualized in its entirety in a last verification pass performed May 2017. In total, 132 products were found which satisfied the conservative selection criteria. Further modifications of the considered objects of research that occurred after May 2017 were not considered and are not reported in this article.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Pool_of_gathered_products\">Pool of gathered products<\/span><\/h3>\n<p>The selected products cover a wide range of categories. The largest categories represented are machine tools (33 products), vehicles (18 products), and robotics (11 products), as well as medical and laboratory equipment (9 products each, Figure 2a). The category of machine tools includes mainly desktop machine tools such as the 3D printer <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ultimaker.com\/\" target=\"_blank\">Ultimaker<\/a> or the laser cutter <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.lasersaur.com\/\" target=\"_blank\">Lasersaur<\/a>. The vehicles category mainly consists of bikes like <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.n55.dk\/MANUALS\/SPACEFRAMEVEHICLES\/spaceframevehicles.html\" target=\"_blank\">XYZ Space Frame Vehicles<\/a> and cars like the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/perma.cc\/96W5-ND74\" target=\"_blank\">Tabby OSVehicle<\/a>. The robotics category is largely represented by humanoid robots developed for teaching or research purposes, such as the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.nimbro.net\/OP\/\" target=\"_blank\">igus Humanoid Open Platform<\/a>. Finally, the medical equipment category covers diverse products like OpenBionics\u2019 <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/openbionics.org\/affordableprosthetichands\" target=\"_blank\">Prosthetic Hand<\/a> or diagnostic equipment such as the echo-stethoscope <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.echopen.org\/\" target=\"_blank\">echOpen<\/a>.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Bonvoisin_JOfOpenHard2017_1-1.png\" class=\"image wiki-link\" data-key=\"50cbed8ca7fd1846a72b98585ec96e27\"><img alt=\"Fig2 Bonvoisin JOfOpenHard2017 1-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0b\/Fig2_Bonvoisin_JOfOpenHard2017_1-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Characterization per product category (<b>a<\/b>), per technology (<b>b<\/b>) and per project status (<b>c<\/b>)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In total, a share of 43% of these (57 products) consist exclusively of non-electronic components. The remaining share of 57% (75 products) also comprise electronic hardware as well as software (Figure 2b). Three quarters of the products are currently developed and\/or marketed; while other products originate from projects which are not active anymore (Figure 2b).\n<\/p><p>The distribution of products per maturity level, depicted in Figure 3, indicates that the majority of the selected products has reached a stage of development that enables systematic replication and production; 70% are in production stages.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Bonvoisin_JOfOpenHard2017_1-1.png\" class=\"image wiki-link\" data-key=\"b1bb8d08ce7342ee31d857779cd37d11\"><img alt=\"Fig3 Bonvoisin JOfOpenHard2017 1-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/e1\/Fig3_Bonvoisin_JOfOpenHard2017_1-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Distribution of products per maturity level<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Overall_openness_of_the_published_documentation\">Overall openness of the published documentation<\/span><\/h3>\n<p>Figure 4 depicts the number of products satisfying the criteria <i>a<\/i> to <i>h<\/i>, as well as the distribution of products along the openness index (OI). The most satisfied criterion is the publication of CAD files (criterion <i>a<\/i>, with 77%, 102\/132) and the least satisfied are the publication of a call for contribution and the editability of assembly instructions (criterion <i>g<\/i> and <i>d<\/i>, with respectively 26% and 25%, 34\/ and 33\/132). In total, 11 products satisfy all criteria [<i>a<\/i>,\u2026,<i>h<\/i>], while 10 products satisfy none of them. The arithmetic mean of the openness index of all products is 4.2 points (median value 4 points). \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Bonvoisin_JOfOpenHard2017_1-1.png\" class=\"image wiki-link\" data-key=\"5792f3e8ee3e81a97287716b469a859f\"><img alt=\"Fig4 Bonvoisin JOfOpenHard2017 1-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/e5\/Fig4_Bonvoisin_JOfOpenHard2017_1-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Number of products satisfying the criteria <i>a<\/i> to <i>h<\/i><\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Figure 5 shows the number of products satisfying the four criteria of openness (<i>T<\/i>, <i>A<\/i>, <i>R<\/i>, and <i>C<\/i>). The most satisfied criterion is transparency (<i>T<\/i>, 77%, 102\/132), respectively followed by commercial usability (<i>C<\/i>, 72%, 95\/132), replicability (<i>R<\/i>, 60%, 80\/132), and accessibility (<i>A<\/i>, 41%, 54\/132). In total, 22 products comply with all the four criteria, while 11 others fulfill none of them. The average number of criteria fulfilled is 2.5 (median value 3).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Bonvoisin_JOfOpenHard2017_1-1.png\" class=\"image wiki-link\" data-key=\"5a1a33a4b02c2f2fc01b6a6c61219980\"><img alt=\"Fig5 Bonvoisin JOfOpenHard2017 1-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/fd\/Fig5_Bonvoisin_JOfOpenHard2017_1-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Number of products satisfying the criteria <i>T<\/i>, <i>A<\/i>, <i>R<\/i>, and <i>C<\/i><\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>This data highlights a certain heterogeneity of practices in the publication of OSH product documentation. The large spectrum of openness observed tends to indicate that the different factors of openness are considered by practitioners more as optional factors rather than mandatory ones. On one side of the spectrum, a few products can be considered as fully open, because they satisfy the four criteria of openness (17%, 22\/132) or get the maximal openness index (11%, 15\/132). On the other side of the spectrum, another few products can be considered as fully closed, as they either do not satisfy any of the four criteria of openness (8%, 10\/132) or get a null openness index (8%, 11\/132). Between those extremes, the large majority of products show diverse profiles of partial openness, without clearly identifiable patterns.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Relative_frequency_of_openness_factors\">Relative frequency of openness factors<\/span><\/h3>\n<p>Results provided by Figure 4 suggest that transparency and commercial usability may be aspects which are given high importance on average or which are easily achievable, while accessibility is an aspect which may be given low importance on average or difficult to achieve.\n<\/p><p>Transparency is not associated with high additional costs, as it is related to the publication of files which are required in the development process (CAD files and schematics). Nonetheless, a significant number of products (almost one fourth) are not providing these files publicly.\n<\/p><p>Commercial usability is related to IP risks. The commercial usability aspect is at the core of the concept of open source, yet more than one third of the products are not provided with a licence allowing commercial usage or are published under licences excluding commercial usage. In the latter case, these products are actually unambiguously not complying with the Open Source Hardware Statement of Principles 1.0, which requires explicitly the use of licences allowing commercial usability.\n<\/p><p>The lower occurrence rate of replicability may be explained by the corresponding additional effort. Indeed, it requires more resources to make a product replicable than transparent or commercially usable. Furthermore, while transparency is about sharing files which are required for the development process of the product and exist whether or not the product is open source, replicability requires the formalization of assembly instructions and bill of materials, which is a time intensive activity. Not all communities may be willing to make the effort to formalize these documents. Moreover, assembly instructions may only be relevant for products which are meant to be produced in a DIY production setting. In the context of projects dedicated to the development of complex OSH products which are meant for industrial production, assembly instructions may be secondary or even irrelevant information.\n<\/p><p>The lower occurrence rate of accessibility may also be explained by the corresponding additional effort. As introduced earlier, accessibility is a precondition for the emergence of a community-based co-development process. Integrating contributions from a development community implies significant coordination costs and requires dedicated resources. As accessibility supports a process which is difficult to implement in practice, not all product originators may give this aspect a high priority.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Role_of_contextual_criteria_in_openness\">Role of contextual criteria in openness<\/span><\/h3>\n<p>Table 2 compares the openness of all products categorized along three assessed contextual criteria: product maturity, whether products are purely mechanical or mechatronic, and whether the surrounding community is active or inactive.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"11\"><b>Table 2.<\/b> Openness vs. contextual criteria\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of products\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">OI\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Transparency\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Accessibility\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Replicability\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Commercial usability\n<\/th><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Mean value\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Absolute\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Relative\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Absolute\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Relative\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Absolute\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Relative\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Absolute\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Relative\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Concept\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.64\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">43%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Prototype\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.54\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">54%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">38%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">62%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/DIY\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">61\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.93\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">54\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">89%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">43%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">48\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">79%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">75%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/kit\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.38\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">94%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">94%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/full product\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.80\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">53%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">67%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mechanic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">57\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.65\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">38\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">67%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">44%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">53%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">68%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mechatronic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">75\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.53\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">64\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">39%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">67%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">56\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">75%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Active\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.57\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">46%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">67\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">68%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">76\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">77%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Inactive\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.91\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">58%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">39%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">58%\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>A first possible reason for the absence of published documentation may be that this documentation does not exist and cannot exist in the level of detail required by the criterion used here. Indeed, in early design stages, the product documentation may not be mature enough for the product concept to be formalized into CAD files, assembly instructions, and bills of materials. This explanation tends to be supported by the gathered data, as the average openness index of products in the early phases (concept and prototype) is significantly lower than those of production phases (DIY, kit, full product production). The percentage of products satisfying the criterion of transparency grows along with product maturity: it is lower than 54% in the concept and prototype phases and grows over 89% in the other phases. The average openness index increases as well; however, it starts to sink again at the last stage (full product production) due to lower replicability and commercial reusability. In this phase, products are marketed by companies which are taking financial risks. This may make them hesitant to disclose information that facilitates imitation.\n<\/p><p>A second possible reason for the absence of published documentation may be that the community which drove the product development is not active anymore. In that case, there is not enough workforce anymore to maintain data online; links become \u201cdead links\u201d and information finally disappears. This explanation tends to be supported by the gathered data, as the average openness index is significantly higher for products with active surrounding communities than for products with inactive communities (respectively 4.57 points and 2.91 points).\n<\/p><p>A third possible reason for the absence of published documentation may be that the community which drives the product development does not follow an open-source approach for the entire product but only for selected components. A straightforward case is when products include off-the-shelf components which are not open source. Another case is when originators deliberately exclude a developed component from the open-source approach in order to protect their strategic knowledge. This behavior has been previously empirically observed and is discussed by Balka, Raasch, and Herstatt.<sup id=\"rdp-ebb-cite_ref-BalkaTheEffect14_5-5\" class=\"reference\"><a href=\"#cite_note-BalkaTheEffect14-5\">[5]<\/a><\/sup> It is however explicitly excluded by the Open Source Hardware Certification Programme of the Open Source Hardware Association, as it is claimed that \u201call parts, designs, code, and rights under the control of the creator must be made open.\u201d<sup id=\"rdp-ebb-cite_ref-OSHAReqs17_52-0\" class=\"reference\"><a href=\"#cite_note-OSHAReqs17-52\">[46]<\/a><\/sup> In the current study, only the openness of mechanical components has been assessed, and those of software and electronic hardware components contained in mechatronic products have been left aside. It is therefore possible that a mechatronic product appears here as non-open although the corresponding electronic hardware and software are open source. Following this explanation, there would be a high chance that the average openness of mechanical parts of mechatronic products should be lower than those of purely mechanical products. This explanation is however not supported by the gathered data, as the average openness index is higher for mechatronic products than for purely mechanical products.\n<\/p><p>The following other possible interpretations of the absence of published documentation cannot be verified by the data gathered in this article:\n<\/p>\n<ul><li> The intention of openness is given in the product development project, but the project lacks capacity to document and to put the product-related documentation online. Although in open-source communities publishing documentation is seen as a way to gather new workforce, generating product documentation remains a time-consuming process and may get lower priority than other product development activities.<\/li>\n<li> There is a certain delay between the statement that the product is open source and the actual disclosure of product-related information. Project initiators may start claiming a product is open source before the corresponding documentation is put online.<\/li>\n<li> The product is claimed to be open source whereas no intention to publish product-related information is given in the product development project, what we can term here as \u201copenwashing\u201d in analogy to \u201cgreenwashing\u201d or \u201cwhitewashing.\u201d<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Typical_profiles\">Typical profiles<\/span><\/h3>\n<p>In order to identify possible typical profiles, the k-medioids algorithm has been computed on the dataset leading to the identification of five clusters. The similarity matrix reordered according to the identified clusters is displayed in Figure 6. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Bonvoisin_JOfOpenHard2017_1-1.png\" class=\"image wiki-link\" data-key=\"07abe8c7583ae4d7c98c16d3f2c26eb0\"><img alt=\"Fig6 Bonvoisin JOfOpenHard2017 1-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/e0\/Fig6_Bonvoisin_JOfOpenHard2017_1-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> Heatmap representing the Manhattan distance Sij between each pair of the 132 products (dark green pairs are similar, light green are dissimilar)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The openness of the products of each cluster is displayed in Table 3 and described hereafter:\n<\/p>\n<ul><li> Cluster 1. Fully open products which satisfy almost all openness criteria and having hence a high average OI value (mean value 6.83\/8)<\/li>\n<li> Cluster 2 and 3. Products being transparent, replicable and commercially usable but not accessible; their average OI value is medium to high (mean value respectively 5.23 and 3.81\/8) and is mainly handicapped by the lack of accessibility.<\/li>\n<li> Cluster 4. Products being transparent, accessible and commercially usable but not replicable; their average OI value is medium to low (mean value 2.94\/8) and is mainly handicapped by the lack of replicability<\/li>\n<li> Cluster 5. Fully closed products, i.e., products for which almost no documentation can be found and out of which two thirds do not provide commercially usable documentation; their average OI value is very low (mean value 0.83\/8)<\/li><\/ul>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"11\"><b>Table 3.<\/b> Openness of the four identified clusters\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Cluster\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of products\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">OI\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Transparency\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Accessibility\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Replicability\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Commercial usability\n<\/th><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Mean value\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Absolute\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Relative\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Absolute\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Relative\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Absolute\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Relative\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Absolute\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Relative\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">C1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.83\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">90%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">C2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">235\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">87%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">C3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.81\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">90%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">77%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">C4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.94\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">59%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">76%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">C5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.83\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33%\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Apart from products showing full or near zero openness (C1 and C5), this clustering displays two other profiles: products developed in the frame of projects thriving at all openness aspects excluding accessibility (C2 and C3), and products developed in the frame of projects thriving at all openness aspects excluding replicability (C4). These results tend to highlight the existence of two OSH project archetypes: the one using openness as a way to support the emergence of collaborative development in communities, the other using openness as a way to support the broad diffusion of the outcome of a conventional and closed product development process. Cluster C1 can be interpreted as a combination of these two archetypes, i.e., as projects of C4 which achieved the product development phase, reached the production and commercialization phase and started to strive for replicability.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>What emerges from the analysed data is a multifaceted picture of OSH which confirms the existence of a \u201cconfusion on what actually makes a project an open source project\u201d identified by Gacek and Arief<sup id=\"rdp-ebb-cite_ref-GacekTheMany04_39-2\" class=\"reference\"><a href=\"#cite_note-GacekTheMany04-39\">[34]<\/a><\/sup> in the field of OSS. Product originators tend to make use of the large room for interpretation given by the fuzzy definition of the term \u201copen source hardware,\u201d to pick up the openness factors best fitting with their situation and therefore to answer for themselves the question \u201cwhat is the source of open source hardware?\u201d in different ways.\n<\/p><p>Whereas none of the four openness factors are represented by more than three fourths of the gathered products, the factors of transparency and commercial usability are nonetheless considered mandatory by most practitioners. The respective non-compliance rates of 23% and 28% may be considered as a normal and reasonable gap between theory and practice. Transparency is the most represented openness factor, and the non-compliance to this factor can be largely explained by the fact that some products are not mature enough for CAD files to be produced and published. Commercial usability can simply be considered as mandatory as it is the only factor of openness explicitly addressed in both the Open Source Definition and the Open Source Hardware Statement of Principles 1.0. Furthermore, it can be viewed as a prerequisite to flank transparency.\n<\/p><p>The somewhat optional nature of the openness factors of accessibility and replicability bases the identification of two main product development project archetypes in OSH. One makes use of open-source publication of product-related documentation as a means to support community-based product development, while the other makes use of these same means for supporting diffusion of their privately developed product. Interestingly, projects of the first group are less numerous than those of the second group. These results are in contradiction with the general use of the term \u201copen source\u201d as a depiction of a collaborative product development process (as stated by Gacek and Arief<sup id=\"rdp-ebb-cite_ref-GacekTheMany04_39-3\" class=\"reference\"><a href=\"#cite_note-GacekTheMany04-39\">[34]<\/a><\/sup>).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Framing_the_.E2.80.9Csource.E2.80.9D_of_open_source_hardware\">Framing the \u201csource\u201d of open source hardware<\/span><\/h3>\n<p>It appears from the reported analysis that the \u201csource\u201d of OSH tends to be interpreted in practice as a dataset whose contents and properties evolve along the product lifecycle. This is summarized in the open-source hardware lifecycle depicted in Figure 7. This original framework defines two states of OSH, characterized by specific motivations and shared product documentation as well as two modes for developing open source hardware products. It also specifies in more detail the concepts which contours have been previously defined in scholarly contributions.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Bonvoisin_JOfOpenHard2017_1-1.png\" class=\"image wiki-link\" data-key=\"fe89102ddb32f52a073942de274e6612\"><img alt=\"Fig7 Bonvoisin JOfOpenHard2017 1-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/37\/Fig7_Bonvoisin_JOfOpenHard2017_1-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 7.<\/b> Open-source hardware lifecycle\n<\/td><\/tr><\/table>\n<\/td><\/tr><\/table>\n<p>Within the product development process, an open-source approach may be used as a means to support the emergence of community-based product development. The OSH product is then before all the object of an open-source product development process (state 1 in Figure 7). In order to be labelled as \u201copen source,\u201d this process requires at least commercial usability, transparency, and accessibility. At early development stages, transparency can only be realized through the publication of descriptive text and simple schematics. Along the development of the product, stable technical drawings emerge (CAD files) which have to be shared in order to support further collaborative development. Accessibility is ensured by the use of editable formats (e.g., native and open CAD formats instead of STL files, native and open text editor files instead of PDF files).\n<\/p><p>Once a product is fully defined and can be produced, it can be documented as an open-source product (state 2 in Figure 7). In this case, open source is used to support the production and diffusion of the product and its durability (through repairability, modifiability and upgradability). In order to be labelled as \u201copen source,\u201d this product can be delivered with additional information, such as bill of materials and assembly instructions, which support production. Making use of this information for actual production and commercialization purposes requires this content to be published with licences allowing commercial usage. During the usage phase and at the product end-of-life, access to this same information may support the aspects of durability mentioned above.\n<\/p><p>An open source product can either be the result of an open-source product development process (mode 1 in Figure 7) or of the disclosing of documentation developed in a private setting\u2014defined as public innovation (mode 2 in Figure 7) by Huizingh.<sup id=\"rdp-ebb-cite_ref-HuizinghOpen11_37-1\" class=\"reference\"><a href=\"#cite_note-HuizinghOpen11-37\">[32]<\/a><\/sup> Note that the latter is not an open-source development approach because it neither supports accessibility nor transparency. Upon the release of an open-source product, the basis is made for new versions to be developed within any one of the two modes. Once a new version of an open-source product is released, it becomes the basis for further continuous open-source product development or public innovation within a private innovation process.\n<\/p><p>The state \u201copen-source product development\u201d has been previously termed in literature as open-source innovation<sup id=\"rdp-ebb-cite_ref-HuizinghOpen11_37-2\" class=\"reference\"><a href=\"#cite_note-HuizinghOpen11-37\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RaaschOnThe09_15-2\" class=\"reference\"><a href=\"#cite_note-RaaschOnThe09-15\">[14]<\/a><\/sup> and as open design.<sup id=\"rdp-ebb-cite_ref-BalkaOpen09_40-2\" class=\"reference\"><a href=\"#cite_note-BalkaOpen09-40\">[35]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AutamurtoTheOpen15_36-1\" class=\"reference\"><a href=\"#cite_note-AutamurtoTheOpen15-36\">[31]<\/a><\/sup> The state \u201copen-source product\u201d is the result of either open-source innovation or public innovation as termed by Huizingh.<sup id=\"rdp-ebb-cite_ref-HuizinghOpen11_37-3\" class=\"reference\"><a href=\"#cite_note-HuizinghOpen11-37\">[32]<\/a><\/sup> The lifecycle model is consequently in contradiction to an exclusive definition of OSH as a subcategory of open-source innovation, as defined by Raasch, Herstatt, and Balka<sup id=\"rdp-ebb-cite_ref-RaaschOnThe09_15-3\" class=\"reference\"><a href=\"#cite_note-RaaschOnThe09-15\">[14]<\/a><\/sup> as well as Huizingh.<sup id=\"rdp-ebb-cite_ref-HuizinghOpen11_37-4\" class=\"reference\"><a href=\"#cite_note-HuizinghOpen11-37\">[32]<\/a><\/sup> This would omit a large part of the actual practices rightly claiming the label of \u201copen-source hardware,\u201d which also includes the free revealing of innovations achieved in private settings. In those cases, however, labeling the product as open source would only make sense after a public release. An open-source product development project as defined here may already fulfil the necessary requirements earlier.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Limits_of_the_reported_research\">Limits of the reported research<\/span><\/h3>\n<p>The list of OSH products gathered for this study cannot be claimed to be exhaustive, and its representativeness for the entire field of OSH has to be considered cautiously. The authors also cannot exclude that the method used to search for eligible products may have omitted subsets of the field. Particularly, product development projects which are in the early development phases may be more difficult to find because they are still poorly documented and their communities are relatively small, if there is any at all yet. This may be the same for projects which acquired low publicity and are therefore poorly ranked in common internet search engines. Nonetheless, the list of OSH products gathered for this study, with this focus, is the largest that has been published so far. At the time the data acquisition campaign was terminated (April 2017), the marginal search time for finding a new project was high enough to justify freezing the dataset since the risk of omitting a significant part of the studied phenomenon was reasonably low.\n<\/p><p>Additionally, the broad field of OSH was narrowed down to discrete, tangible, non-electronic and complex products, which excludes a large part of the field (such as millions of gimmicks). In terms of how far the results discussed in this article are valid for the whole range of OSH products, including low complexity products as well as purely electronic products, remains an open question. Further research could make use of the methods defined in this article in order to perform a similar analysis specifically addressing electronic hardware. To the knowledge of the authors, no such study has been published so far. The OSHWA maintains a list of certified OSH products, most of which are electronic products, providing therefore an interesting data basis.\n<\/p><p>The assessment of the corresponding published product-related documentation has been simplified to the evaluation of binary criteria. Each product was assessed as to whether certain types of documents are provided. The quality of the published documentation with regard to the level of detail, comprehensiveness, or clarity has not been examined. Do published CAD files represent the whole product or just parts of it? Are the guidelines for participation easily understandable for potential participants and provide them with the right information? What information is displayed in parts lists and in what form? These questions were not taken into account because of the need to reduce the data acquisition effort to a manageable level. This simplification may have produced a positive bias, whereby products may have been rated more open than they are. Future research is needed in order to define to what extent published product-related documentation is actually usable and useful.\n<\/p><p>On the other hand, the practical cut-off rule for each binary criterion may have introduced a negative bias. Products may have been rated less open than they are. Simply because some pieces of documentation have not been found after 10 minutes of internet search does not necessarily mean that information cannot be found online. The cut-off rule applied is a practical interpretation of the Open Source Definition which requires that documents can be accessed \u201ceasily.\u201d Another negative bias comes from the fact that only information related to the non-electronic hardware components included in the respective products was evaluated. Documentation regarding electronic hardware and software has not been regarded as a practical cut-off in data acquisition (see the subsection on \u201cData acquisition\u201d), hence, parts of openness have been omitted. Needless to say, the openness of electronic hardware is as important as those of other types of hardware. Consequently, the openness of a mechatronic product should be assessed upon those of its electronic and mechanical components.\n<\/p><p>The dataset on which this article is based upon has been published with a CC-BY licence.<sup id=\"rdp-ebb-cite_ref-BonvoisinOpenness17_53-0\" class=\"reference\"><a href=\"#cite_note-BonvoisinOpenness17-53\">[47]<\/a><\/sup> An actualized version of this dataset is available under the same licensing terms in the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/opensourcedesign.cc\/observatory\" target=\"_blank\">Open Source Hardware Directory<\/a>. This database of complex OSH products allows any interested person to add references of OSH products, or to edit existing ones. The content of this database can be used for any purpose, such as future research, as suggested in this article. The objective of the methodology pursued in this article and in the generation of the data was to generate an overview of the field of OSH. The authors cannot guarantee that the presented data of more than 2000 items is free of errors. Should the originators or contributors to the OSH products referenced in this dataset consider their products misrepresented, the authors kindly invite them to make corrections. They are free to edit the provided database or inform the authors of detected errors.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>This article provides an overview of how the concept of OSH is interpreted in practice, in other words, what is the \u201csource\u201d of OSH for practitioners. Starting from a critical analysis of existing definitions, evaluation criteria have been defined and used to analyse the published documentation of a pool of 132 complex and non-electronic OSH products.\n<\/p><p>The gathered data illustrates how the field of OSH has moved well beyond the scope of electronic hardware and towards the field of non-electronic complex product development. The reported research required the establishment of a database of OSH products, which is the largest published to date. This database provides empirical evidence of the evolution of OSH outside the sphere of electronics, a phenomenon which presently has only been described by scholarly publications based on isolated cases.\n<\/p><p>The analysis of the published documentation of the 132 identified OSH products confirms the wide range of documentation sharing practices which may also be due to the fuzzy definitions of OSH. Product documentation ranges from very demanding to very lax compliance, i.e., from the mere sharing of CAD files to the publication of comprehensive sets of documents. The results highlight a surprisingly large share of products which are provided with barely any information at all (cluster C5, 18%). This could either be due to a misinterpretation of the concept of OSH, a deliberate intention to \u201copenwash\u201d a product, or the influence of the time variable. What can be understood from this is that there is a need to set standards in order to achieve clarity in the field of OSH. This will support the emergence of a constructive public discourse around these new practices.\n<\/p><p>Clustering the characteristics of published product-related documentation led to the identification of two distinct states of OSH. These have been understood as resulting from differing though not mutually exclusive strategic modes: namely the stimulation of community-based product development and the public dissemination of privately achieved innovation. The interaction between those states has been summarized in the definition of an OSH lifecycle summarizing observed approaches to OSH. It is the hope of the authors that this article has introduced solid categories that have provided a foundation on which further practice and research activities can be based.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Footnotes\">Footnotes<\/span><\/h2>\n<div class=\"reflist\" style=\"list-style-type: lower-alpha;\">\n<ol class=\"references\">\n<li id=\"cite_note-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-7\">\u2191<\/a><\/span> <span class=\"reference-text\">For an overview of OSH licences, see for example Katz.<sup id=\"rdp-ebb-cite_ref-KatzTowards12_6-0\" class=\"reference\"><a href=\"#cite_note-KatzTowards12-6\">[6]<\/a><\/sup><\/span>\n<\/li>\n<li id=\"cite_note-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-17\">\u2191<\/a><\/span> <span class=\"reference-text\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.thingiverse.com\/\" target=\"_blank\">Thingiverse<\/a> is a platform that allows users to share a huge variety of 3D models based on CAD or STL files.<\/span>\n<\/li>\n<li id=\"cite_note-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-18\">\u2191<\/a><\/span> <span class=\"reference-text\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/perma.cc\/XZ7W-ELAQ\" target=\"_blank\">Enable Outreach<\/a> is a network of makers that collaborate locally with disabled children to provide them with adapted 3D printed hand prosthesis.<\/span>\n<\/li>\n<li id=\"cite_note-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-28\">\u2191<\/a><\/span> <span class=\"reference-text\">Evolutionary design may not generally contradict parallelization or staging of processes, i.e., in the intra-project environment for scaling and efficiency. Also, this is different from the predetermined mechanism of breaking down problems into constituent parts (modularization) and avoids creation of task interdependencies.<\/span>\n<\/li>\n<li id=\"cite_note-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-31\">\u2191<\/a><\/span> <span class=\"reference-text\">These four freedoms are derived from the free software definition<sup id=\"rdp-ebb-cite_ref-GNUWhatIs15_30-0\" class=\"reference\"><a href=\"#cite_note-GNUWhatIs15-30\">[26]<\/a><\/sup>, which are: Freedom 0, the freedom to run the program for any purpose; Freedom 1, the freedom to study how the program works; Freedom 2, the freedom to redistribute copies; Freedom 3, the freedom to distribute copies of modified versions. In the transition from the context of immaterial intellectual property to the realm of tangible products, these freedoms have been reinterpreted. For example, running a program requires compiling the source code (an action alternatively termed as build or make in the software jargon). The freedom to run became the freedom to make the product, that is, to produce it.<\/span>\n<\/li>\n<li id=\"cite_note-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-41\">\u2191<\/a><\/span> <span class=\"reference-text\">or deeper insights into the different degrees of involvement in open source communities, see for example Gacek and Arief.<sup id=\"rdp-ebb-cite_ref-GacekTheMany04_39-1\" class=\"reference\"><a href=\"#cite_note-GacekTheMany04-39\">[34]<\/a><\/sup><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The reported research has been performed in the frame of the French-German interdisciplinary research project \u201cOpen! \u2013 Methods and tools for community-based product development.\u201d It is jointly funded by the French and German national science agencies ANR (Agence Nationale de la Recherche, grant ANR-15-CE26-0012) and DFG (Deutsche Forschungsgemeinschaft, grants JO 827\/8-1 and STA 1112\/13-1). The authors would like thank the anonymous reviewers for their insightful suggestions and constructive feedback. Last but not least, they would like to acknowledge Kerstin Carola Schmidt for her contributions to the data collection and assessment.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors have no competing interests to declare.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HatchTheMaker13-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HatchTheMaker13_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Hatch, M. (2013). <i>The Maker Movement Manifesto: Rules for Innovation in the New World of Crafters, Hackers, and Tinkerers<\/i>. McGraw-Hill Education. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780071821124.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Maker+Movement+Manifesto%3A+Rules+for+Innovation+in+the+New+World+of+Crafters%2C+Hackers%2C+and+Tinkerers&rft.aulast=Hatch%2C+M.&rft.au=Hatch%2C+M.&rft.date=2013&rft.pub=McGraw-Hill+Education&rft.isbn=9780071821124&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VoigtAnEmp16-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VoigtAnEmp16_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Voigt, C.; Montero, C.S.; Menichinelli, M. (2016). \"An Empirically Informed Taxonomy for the Maker Movement\". In Bagnoli, F., Satsiou, A.; Stavrakakis, I. et al.. <i>Internet Science - INSCI 2016<\/i>. Lecture Notes in Computer Science. <b>9934<\/b>. Springer. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-45982-0_17\" target=\"_blank\">10.1007\/978-3-319-45982-0_17<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783319459820.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=An+Empirically+Informed+Taxonomy+for+the+Maker+Movement&rft.atitle=Internet+Science+-+INSCI+2016&rft.aulast=Voigt%2C+C.%3B+Montero%2C+C.S.%3B+Menichinelli%2C+M.&rft.au=Voigt%2C+C.%3B+Montero%2C+C.S.%3B+Menichinelli%2C+M.&rft.date=2016&rft.series=Lecture+Notes+in+Computer+Science&rft.volume=9934&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-319-45982-0_17&rft.isbn=9783319459820&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BalkaOpen11-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BalkaOpen11_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Balka, K. (2011). <i>Open Source Product Development - The Meaning an Relevance of Openness<\/i>. Gabler Verlag. p. 4. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-8349-6949-1\" target=\"_blank\">10.1007\/978-3-8349-6949-1<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783834969491.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Open+Source+Product+Development+-+The+Meaning+an+Relevance+of+Openness&rft.aulast=Balka%2C+K.&rft.au=Balka%2C+K.&rft.date=2011&rft.pages=p.%26nbsp%3B4&rft.pub=Gabler+Verlag&rft_id=info:doi\/10.1007%2F978-3-8349-6949-1&rft.isbn=9783834969491&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RaaschProduct11-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RaaschProduct11_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-RaaschProduct11_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-RaaschProduct11_4-2\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Raasch, C. (2011). \"Product Development in Open Design Communities: A Process Perspective\". <i>International Journal of Innovation and Technology Management<\/i> <b>8<\/b> (4): 557\u201375. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1142%2FS021987701100260X\" target=\"_blank\">10.1142\/S021987701100260X<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Product+Development+in+Open+Design+Communities%3A+A+Process+Perspective&rft.jtitle=International+Journal+of+Innovation+and+Technology+Management&rft.aulast=Raasch%2C+C.&rft.au=Raasch%2C+C.&rft.date=2011&rft.volume=8&rft.issue=4&rft.pages=557%E2%80%9375&rft_id=info:doi\/10.1142%2FS021987701100260X&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BalkaTheEffect14-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BalkaTheEffect14_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-BalkaTheEffect14_5-1\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-BalkaTheEffect14_5-2\">5.2<\/a><\/sup> <sup><a href=\"#cite_ref-BalkaTheEffect14_5-3\">5.3<\/a><\/sup> <sup><a href=\"#cite_ref-BalkaTheEffect14_5-4\">5.4<\/a><\/sup> <sup><a href=\"#cite_ref-BalkaTheEffect14_5-5\">5.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Balka, K.; Raasch, C.; Herstatt, C. (2014). \"The Effect of Selective Openness on Value Creation in User Innovation Communities\". <i>The Journal of Product Innovation Management<\/i> <b>31<\/b> (2): 392\u2013407. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjpim.12102\" target=\"_blank\">10.1111\/jpim.12102<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Effect+of+Selective+Openness+on+Value+Creation+in+User+Innovation+Communities&rft.jtitle=The+Journal+of+Product+Innovation+Management&rft.aulast=Balka%2C+K.%3B+Raasch%2C+C.%3B+Herstatt%2C+C.&rft.au=Balka%2C+K.%3B+Raasch%2C+C.%3B+Herstatt%2C+C.&rft.date=2014&rft.volume=31&rft.issue=2&rft.pages=392%E2%80%93407&rft_id=info:doi\/10.1111%2Fjpim.12102&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KatzTowards12-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KatzTowards12_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Katz, A. (2012). \"Towards a Functional Licence for Open Hardware\". <i>The Journal of Open Law, Technology & Society<\/i> <b>4<\/b> (1): 41\u201362. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5033%2Fifosslr.v4i1.69\" target=\"_blank\">10.5033\/ifosslr.v4i1.69<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+a+Functional+Licence+for+Open+Hardware&rft.jtitle=The+Journal+of+Open+Law%2C+Technology+%26+Society&rft.aulast=Katz%2C+A.&rft.au=Katz%2C+A.&rft.date=2012&rft.volume=4&rft.issue=1&rft.pages=41%E2%80%9362&rft_id=info:doi\/10.5033%2Fifosslr.v4i1.69&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BalkaOpen16-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BalkaOpen16_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Balka, K. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20160927222446\/http:\/\/open-innovation-projects.org\/project-list\" target=\"_blank\">\"Project List\"<\/a>. <i>Open Innovation Projects<\/i>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/open-innovation-projects.org\/project-list\" target=\"_blank\">the original<\/a> on 27 September 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20160927222446\/http:\/\/open-innovation-projects.org\/project-list\" target=\"_blank\">https:\/\/web.archive.org\/web\/20160927222446\/http:\/\/open-innovation-projects.org\/project-list<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Project+List&rft.atitle=Open+Innovation+Projects&rft.aulast=Balka%2C+K.&rft.au=Balka%2C+K.&rft.date=2016&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20160927222446%2Fhttp%3A%2F%2Fopen-innovation-projects.org%2Fproject-list&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoritzOpen15-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoritzOpen15_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moritz, M.; Redlich, T.; Krenz, P. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.iiisci.org\/journal\/sci\/FullText.asp?var=&id=HB188KY15\" target=\"_blank\">\"Open up or Close down - The new Era of \u201cOpenneers\u201d and how they lead the Way to Future Success\"<\/a>. <i>Journal of Systemics, Cybernetics and Informatics<\/i> <b>13<\/b> (6): 15\u201322<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.iiisci.org\/journal\/sci\/FullText.asp?var=&id=HB188KY15\" target=\"_blank\">http:\/\/www.iiisci.org\/journal\/sci\/FullText.asp?var=&id=HB188KY15<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+up+or+Close+down+-+The+new+Era+of+%E2%80%9COpenneers%E2%80%9D+and+how+they+lead+the+Way+to+Future+Success&rft.jtitle=Journal+of+Systemics%2C+Cybernetics+and+Informatics&rft.aulast=Moritz%2C+M.%3B+Redlich%2C+T.%3B+Krenz%2C+P.+et+al.&rft.au=Moritz%2C+M.%3B+Redlich%2C+T.%3B+Krenz%2C+P.+et+al.&rft.date=2015&rft.volume=13&rft.issue=6&rft.pages=15%E2%80%9322&rft_id=http%3A%2F%2Fwww.iiisci.org%2Fjournal%2Fsci%2FFullText.asp%3Fvar%3D%26id%3DHB188KY15&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PearceEmerging17-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PearceEmerging17_10-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-PearceEmerging17_10-1\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pearce, J.M. (2017). \"Emerging Business Models for Open Source Hardware\". <i>Journal of Open Hardware<\/i> <b>1<\/b> (1): 2. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5334%2Fjoh.4\" target=\"_blank\">10.5334\/joh.4<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Emerging+Business+Models+for+Open+Source+Hardware&rft.jtitle=Journal+of+Open+Hardware&rft.aulast=Pearce%2C+J.M.&rft.au=Pearce%2C+J.M.&rft.date=2017&rft.volume=1&rft.issue=1&rft.pages=2&rft_id=info:doi\/10.5334%2Fjoh.4&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiWhyOpen17-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiWhyOpen17_11-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiWhyOpen17_11-1\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, Z.; Seering, W.; Ramos, J.D. et al. (2017). \"Why Open Source?: Exploring the Motivations of Using an Open Model for Hardware Development\". <i>Proceedings of the ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1115%2FDETC2017-68195\" target=\"_blank\">10.1115\/DETC2017-68195<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Why+Open+Source%3F%3A+Exploring+the+Motivations+of+Using+an+Open+Model+for+Hardware+Development&rft.jtitle=Proceedings+of+the+ASME+2017+International+Design+Engineering+Technical+Conferences+and+Computers+and+Information+in+Engineering+Conference&rft.aulast=Li%2C+Z.%3B+Seering%2C+W.%3B+Ramos%2C+J.D.+et+al.&rft.au=Li%2C+Z.%3B+Seering%2C+W.%3B+Ramos%2C+J.D.+et+al.&rft.date=2017&rft_id=info:doi\/10.1115%2FDETC2017-68195&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FjeldstedOpen12-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FjeldstedOpen12_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fjeldsted, A.S.; Adalsteinsdottir, G.; Howard, T.J. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/orbit.dtu.dk\/en\/publications\/open-source-development-of-tangible-products-from-a-business-pers\" target=\"_blank\">\"Open Source Development of Tangible Products - From a business perspective\"<\/a>. <i>Proceedings of NordDesign 2012<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/orbit.dtu.dk\/en\/publications\/open-source-development-of-tangible-products-from-a-business-pers\" target=\"_blank\">https:\/\/orbit.dtu.dk\/en\/publications\/open-source-development-of-tangible-products-from-a-business-pers<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+Source+Development+of+Tangible+Products+-+From+a+business+perspective&rft.jtitle=Proceedings+of+NordDesign+2012&rft.aulast=Fjeldsted%2C+A.S.%3B+Adalsteinsdottir%2C+G.%3B+Howard%2C+T.J.+et+al.&rft.au=Fjeldsted%2C+A.S.%3B+Adalsteinsdottir%2C+G.%3B+Howard%2C+T.J.+et+al.&rft.date=2012&rft_id=https%3A%2F%2Forbit.dtu.dk%2Fen%2Fpublications%2Fopen-source-development-of-tangible-products-from-a-business-pers&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BonvoisinOpen15-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BonvoisinOpen15_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bonvoisin, J.; Boujut, J.-F. (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.designsociety.org\/publication\/37899\/OPEN+DESIGN+PLATFORMS+FOR+OPEN+SOURCE+PRODUCT+DEVELOPMENT%3A+CURRENT+STATE+AND+REQUIREMENTS\" target=\"_blank\">\"Open Design Platforms for Open Source Product Development: Current State and Requirements\"<\/a>. <i>Proceedings of the 20th International Conference on Engineering Design<\/i> <b>8<\/b>: 11\u201320<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.designsociety.org\/publication\/37899\/OPEN+DESIGN+PLATFORMS+FOR+OPEN+SOURCE+PRODUCT+DEVELOPMENT%3A+CURRENT+STATE+AND+REQUIREMENTS\" target=\"_blank\">https:\/\/www.designsociety.org\/publication\/37899\/OPEN+DESIGN+PLATFORMS+FOR+OPEN+SOURCE+PRODUCT+DEVELOPMENT%3A+CURRENT+STATE+AND+REQUIREMENTS<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+Design+Platforms+for+Open+Source+Product+Development%3A+Current+State+and+Requirements&rft.jtitle=Proceedings+of+the+20th+International+Conference+on+Engineering+Design&rft.aulast=Bonvoisin%2C+J.%3B+Boujut%2C+J.-F.&rft.au=Bonvoisin%2C+J.%3B+Boujut%2C+J.-F.&rft.date=2015&rft.volume=8&rft.pages=11%E2%80%9320&rft_id=https%3A%2F%2Fwww.designsociety.org%2Fpublication%2F37899%2FOPEN%2BDESIGN%2BPLATFORMS%2BFOR%2BOPEN%2BSOURCE%2BPRODUCT%2BDEVELOPMENT%253A%2BCURRENT%2BSTATE%2BAND%2BREQUIREMENTS&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HowardOpen12-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HowardOpen12_14-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-HowardOpen12_14-1\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Howard, T.J.; Achiche, S.; \u00d6zkil, A. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.designsociety.org\/publication\/31985\/OPEN+DESIGN+AND+CROWDSOURCING%3A+MATURITY%2C+METHODOLOGY+AND+BUSINESS+MODELS\" target=\"_blank\">\"Open Design and Crowdsourcing: Maturity, Methodology and Business Models\"<\/a>. <i>Proceedings of DESIGN 2012<\/i>: 181\u201390<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.designsociety.org\/publication\/31985\/OPEN+DESIGN+AND+CROWDSOURCING%3A+MATURITY%2C+METHODOLOGY+AND+BUSINESS+MODELS\" target=\"_blank\">https:\/\/www.designsociety.org\/publication\/31985\/OPEN+DESIGN+AND+CROWDSOURCING%3A+MATURITY%2C+METHODOLOGY+AND+BUSINESS+MODELS<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+Design+and+Crowdsourcing%3A+Maturity%2C+Methodology+and+Business+Models&rft.jtitle=Proceedings+of+DESIGN+2012&rft.aulast=Howard%2C+T.J.%3B+Achiche%2C+S.%3B+%C3%96zkil%2C+A.+et+al.&rft.au=Howard%2C+T.J.%3B+Achiche%2C+S.%3B+%C3%96zkil%2C+A.+et+al.&rft.date=2012&rft.pages=181%E2%80%9390&rft_id=https%3A%2F%2Fwww.designsociety.org%2Fpublication%2F31985%2FOPEN%2BDESIGN%2BAND%2BCROWDSOURCING%253A%2BMATURITY%252C%2BMETHODOLOGY%2BAND%2BBUSINESS%2BMODELS&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RaaschOnThe09-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RaaschOnThe09_15-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-RaaschOnThe09_15-1\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-RaaschOnThe09_15-2\">14.2<\/a><\/sup> <sup><a href=\"#cite_ref-RaaschOnThe09_15-3\">14.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Raasch, C.; Herstatt, C.; Balka, K. (2009). \"On the open design of tangible goods\". <i>R&D Management<\/i> <b>39<\/b> (4): 382\u201393. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1467-9310.2009.00567.x\" target=\"_blank\">10.1111\/j.1467-9310.2009.00567.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=On+the+open+design+of+tangible+goods&rft.jtitle=R%26D+Management&rft.aulast=Raasch%2C+C.%3B+Herstatt%2C+C.%3B+Balka%2C+K.&rft.au=Raasch%2C+C.%3B+Herstatt%2C+C.%3B+Balka%2C+K.&rft.date=2009&rft.volume=39&rft.issue=4&rft.pages=382%E2%80%9393&rft_id=info:doi\/10.1111%2Fj.1467-9310.2009.00567.x&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HansenTheCurrent13-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HansenTheCurrent13_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hansen, A.; Howard, T.J. (2013). \"The Current State of Open Source Hardware: The Need for an Open Source Development Platform\". <i>ICoRD'13<\/i>: 977\u201388. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-81-322-1050-4_77\" target=\"_blank\">10.1007\/978-81-322-1050-4_77<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Current+State+of+Open+Source+Hardware%3A+The+Need+for+an+Open+Source+Development+Platform&rft.jtitle=ICoRD%2713&rft.aulast=Hansen%2C+A.%3B+Howard%2C+T.J.&rft.au=Hansen%2C+A.%3B+Howard%2C+T.J.&rft.date=2013&rft.pages=977%E2%80%9388&rft_id=info:doi\/10.1007%2F978-81-322-1050-4_77&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Camarinha-MatosCollab09-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Camarinha-MatosCollab09_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Camarinha,-Matos, L.M.; Afsarmanesh, H.; Galeano, N. et al. (2009). \"Collaborative networked organizations \u2013 Concepts and practice in manufacturing enterprises\". <i>Computers & Industrial Engineering<\/i> <b>57 }issue=1<\/b>: 46\u201360. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cie.2008.11.024\" target=\"_blank\">10.1016\/j.cie.2008.11.024<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Collaborative+networked+organizations+%E2%80%93+Concepts+and+practice+in+manufacturing+enterprises&rft.jtitle=Computers+%26+Industrial+Engineering&rft.aulast=Camarinha%2C-Matos%2C+L.M.%3B+Afsarmanesh%2C+H.%3B+Galeano%2C+N.+et+al.&rft.au=Camarinha%2C-Matos%2C+L.M.%3B+Afsarmanesh%2C+H.%3B+Galeano%2C+N.+et+al.&rft.date=2009&rft.volume=57+%7Dissue%3D1&rft.pages=46%E2%80%9360&rft_id=info:doi\/10.1016%2Fj.cie.2008.11.024&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaherScaling10-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaherScaling10_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maher, M.L.; Paulini, M.; Murty, P. (2010). \"Scaling Up: From Individual Design to Collaborative Design to Collective Design\". <i>Proceedings from Design Computing and Cognition '10<\/i>: 581\u201399. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-94-007-0510-4_31\" target=\"_blank\">10.1007\/978-94-007-0510-4_31<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Scaling+Up%3A+From+Individual+Design+to+Collaborative+Design+to+Collective+Design&rft.jtitle=Proceedings+from+Design+Computing+and+Cognition+%2710&rft.aulast=Maher%2C+M.L.%3B+Paulini%2C+M.%3B+Murty%2C+P.&rft.au=Maher%2C+M.L.%3B+Paulini%2C+M.%3B+Murty%2C+P.&rft.date=2010&rft.pages=581%E2%80%9399&rft_id=info:doi\/10.1007%2F978-94-007-0510-4_31&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PauliniDesign13-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PauliniDesign13_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Paulini, M.; Murty, P.; Maher, M.L. (2013). \"Design processes in collective innovation communities: A study of communication\". <i>CoDesign - International Journal of CoCreation in Design and the Arts<\/i> <b>9<\/b> (2): 90\u2013112. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F15710882.2012.716850\" target=\"_blank\">10.1080\/15710882.2012.716850<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Design+processes+in+collective+innovation+communities%3A+A+study+of+communication&rft.jtitle=CoDesign+-+International+Journal+of+CoCreation+in+Design+and+the+Arts&rft.aulast=Paulini%2C+M.%3B+Murty%2C+P.%3B+Maher%2C+M.L.&rft.au=Paulini%2C+M.%3B+Murty%2C+P.%3B+Maher%2C+M.L.&rft.date=2013&rft.volume=9&rft.issue=2&rft.pages=90%E2%80%93112&rft_id=info:doi\/10.1080%2F15710882.2012.716850&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WestGetting08-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WestGetting08_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">West, J.; Lakhani, K.R. (2008). \"Getting Clear About Communities in Open Innovation\". <i>Industry and Innovation<\/i>: 223\u201331. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F13662710802033734\" target=\"_blank\">10.1080\/13662710802033734<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Getting+Clear+About+Communities+in+Open+Innovation&rft.jtitle=Industry+and+Innovation&rft.aulast=West%2C+J.%3B+Lakhani%2C+K.R.&rft.au=West%2C+J.%3B+Lakhani%2C+K.R.&rft.date=2008&rft.pages=223%E2%80%9331&rft_id=info:doi\/10.1080%2F13662710802033734&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-vonHippelDemoc06-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-vonHippelDemoc06_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">von Hippel, E. (2006). <i>Democratizing Innovation<\/i>. MIT Press. pp. 4, 165. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780262720472.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Democratizing+Innovation&rft.aulast=von+Hippel%2C+E.&rft.au=von+Hippel%2C+E.&rft.date=2006&rft.pages=pp.%26nbsp%3B4%2C+165&rft.pub=MIT+Press&rft.isbn=9780262720472&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AksuluAComp10-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AksuluAComp10_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aksulu, A.; Wade, M.R. (2010). \"A Comprehensive Review and Synthesis of Open Source Research\". <i>Journal of the Association for Information Systems<\/i> <b>11<\/b> (11): 576\u2013656. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.17705%2F1jais.00245\" target=\"_blank\">10.17705\/1jais.00245<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Comprehensive+Review+and+Synthesis+of+Open+Source+Research&rft.jtitle=Journal+of+the+Association+for+Information+Systems&rft.aulast=Aksulu%2C+A.%3B+Wade%2C+M.R.&rft.au=Aksulu%2C+A.%3B+Wade%2C+M.R.&rft.date=2010&rft.volume=11&rft.issue=11&rft.pages=576%E2%80%93656&rft_id=info:doi\/10.17705%2F1jais.00245&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeeFromAFirm03-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeeFromAFirm03_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lee, G.K.; Cole, R.E. (2003). \"From a Firm-Based to a Community-Based Model of Knowledge Creation: The Case of the Linux Kernel Development\". <i>Organization Science<\/i> <b>14<\/b> (6): 615\u2013758. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Forsc.14.6.633.24866\" target=\"_blank\">10.1287\/orsc.14.6.633.24866<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+a+Firm-Based+to+a+Community-Based+Model+of+Knowledge+Creation%3A+The+Case+of+the+Linux+Kernel+Development&rft.jtitle=Organization+Science&rft.aulast=Lee%2C+G.K.%3B+Cole%2C+R.E.&rft.au=Lee%2C+G.K.%3B+Cole%2C+R.E.&rft.date=2003&rft.volume=14&rft.issue=6&rft.pages=615%E2%80%93758&rft_id=info:doi\/10.1287%2Forsc.14.6.633.24866&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HowisonCollab14-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HowisonCollab14_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Howison, J.; Crowston, K. (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/misq.org\/collaboration-through-open-superposition.html\" target=\"_blank\">\"Collaboration Through Open Superposition: A Theory of the Open Source Way\"<\/a>. <i>MIS Quarterly<\/i> <b>38<\/b> (1): 29\u201350<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/misq.org\/collaboration-through-open-superposition.html\" target=\"_blank\">https:\/\/misq.org\/collaboration-through-open-superposition.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Collaboration+Through+Open+Superposition%3A+A+Theory+of+the+Open+Source+Way&rft.jtitle=MIS+Quarterly&rft.aulast=Howison%2C+J.%3B+Crowston%2C+K.&rft.au=Howison%2C+J.%3B+Crowston%2C+K.&rft.date=2014&rft.volume=38&rft.issue=1&rft.pages=29%E2%80%9350&rft_id=https%3A%2F%2Fmisq.org%2Fcollaboration-through-open-superposition.html&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaherForm96-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaherForm96_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maher, M.L.; Poon, J.; Boulanger, S. (1996). \"Formalising Design Exploration as Co-Evolution\". <i>Advances in Formal Design Methods for CAD<\/i>: 3\u201330. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-0-387-34925-1_1\" target=\"_blank\">10.1007\/978-0-387-34925-1_1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Formalising+Design+Exploration+as+Co-Evolution&rft.jtitle=Advances+in+Formal+Design+Methods+for+CAD&rft.aulast=Maher%2C+M.L.%3B+Poon%2C+J.%3B+Boulanger%2C+S.&rft.au=Maher%2C+M.L.%3B+Poon%2C+J.%3B+Boulanger%2C+S.&rft.date=1996&rft.pages=3%E2%80%9330&rft_id=info:doi\/10.1007%2F978-0-387-34925-1_1&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSHADefinition16-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OSHADefinition16_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Open Source Hardware Association (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.oshwa.org\/definition\/\" target=\"_blank\">\"Definition (English)\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.oshwa.org\/definition\/\" target=\"_blank\">https:\/\/www.oshwa.org\/definition\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Definition+%28English%29&rft.atitle=&rft.aulast=Open+Source+Hardware+Association&rft.au=Open+Source+Hardware+Association&rft.date=2016&rft_id=https%3A%2F%2Fwww.oshwa.org%2Fdefinition%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GNUWhatIs15-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GNUWhatIs15_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Free Software Foundation (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.gnu.org\/philosophy\/free-sw.en.html\" target=\"_blank\">\"What is free software?\"<\/a>. <i>GNU.org<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.gnu.org\/philosophy\/free-sw.en.html\" target=\"_blank\">https:\/\/www.gnu.org\/philosophy\/free-sw.en.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+is+free+software%3F&rft.atitle=GNU.org&rft.aulast=Free+Software+Foundation&rft.au=Free+Software+Foundation&rft.date=2015&rft_id=https%3A%2F%2Fwww.gnu.org%2Fphilosophy%2Ffree-sw.en.html&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSITheOpen07-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OSITheOpen07_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Open Source Initiative (2007). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/opensource.org\/osd-annotated\" target=\"_blank\">\"The Open Source Definition (Annotated)\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/opensource.org\/osd-annotated\" target=\"_blank\">https:\/\/opensource.org\/osd-annotated<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Open+Source+Definition+%28Annotated%29&rft.atitle=&rft.aulast=Open+Source+Initiative&rft.au=Open+Source+Initiative&rft.date=2007&rft_id=https%3A%2F%2Fopensource.org%2Fosd-annotated&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BonvoisinBestPrac17-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BonvoisinBestPrac17_33-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-BonvoisinBestPrac17_33-1\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bonvoisin, J.; Schmidt, K.C. (15 February 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/depositonce.tu-berlin.de\/handle\/11303\/6164\" target=\"_blank\">\"Best practices of open source mechanical hardware: A guide with practical advice for sharing product-related documentation\"<\/a>. <i>DepositOnce<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.14279%2Fdepositonce-5729\" target=\"_blank\">10.14279\/depositonce-5729<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/depositonce.tu-berlin.de\/handle\/11303\/6164\" target=\"_blank\">https:\/\/depositonce.tu-berlin.de\/handle\/11303\/6164<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Best+practices+of+open+source+mechanical+hardware%3A+A+guide+with+practical+advice+for+sharing+product-related+documentation&rft.atitle=DepositOnce&rft.aulast=Bonvoisin%2C+J.%3B+Schmidt%2C+K.C.&rft.au=Bonvoisin%2C+J.%3B+Schmidt%2C+K.C.&rft.date=15+February+2017&rft_id=info:doi\/10.14279%2Fdepositonce-5729&rft_id=https%3A%2F%2Fdepositonce.tu-berlin.de%2Fhandle%2F11303%2F6164&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangTheRet12-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangTheRet12_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, H.; Johnson, A.L.; Bracewell, R.H. (2012). \"The retrieval of structured design rationale for the re-use of design knowledge with an integrated representation\". <i>Advanced Engineering Informatics<\/i> <b>26<\/b> (2): 251\u201366. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2012.02.003\" target=\"_blank\">10.1016\/j.aei.2012.02.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+retrieval+of+structured+design+rationale+for+the+re-use+of+design+knowledge+with+an+integrated+representation&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Wang%2C+H.%3B+Johnson%2C+A.L.%3B+Bracewell%2C+R.H.&rft.au=Wang%2C+H.%3B+Johnson%2C+A.L.%3B+Bracewell%2C+R.H.&rft.date=2012&rft.volume=26&rft.issue=2&rft.pages=251%E2%80%9366&rft_id=info:doi\/10.1016%2Fj.aei.2012.02.003&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-vonHippelComment10-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-vonHippelComment10_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">von Hippel, E. (2010). \"Comment on \u2018Is open innovation a field of study or a communication barrier to theory development?\u2019\". <i>Technovation<\/i> <b>30<\/b> (11\u201312): 555. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.technovation.2010.09.003\" target=\"_blank\">10.1016\/j.technovation.2010.09.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comment+on+%E2%80%98Is+open+innovation+a+field+of+study+or+a+communication+barrier+to+theory+development%3F%E2%80%99&rft.jtitle=Technovation&rft.aulast=von+Hippel%2C+E.&rft.au=von+Hippel%2C+E.&rft.date=2010&rft.volume=30&rft.issue=11%E2%80%9312&rft.pages=555&rft_id=info:doi\/10.1016%2Fj.technovation.2010.09.003&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AutamurtoTheOpen15-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AutamurtoTheOpen15_36-0\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-AutamurtoTheOpen15_36-1\">31.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aitamurto, T.; Holland, D.; Hussain, S. (2015). \"The Open Paradigm in Design Research\". <i>Design Issues<\/i> <b>31<\/b> (4): 17\u201329. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1162%2FDESI_a_00348\" target=\"_blank\">10.1162\/DESI_a_00348<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Open+Paradigm+in+Design+Research&rft.jtitle=Design+Issues&rft.aulast=Aitamurto%2C+T.%3B+Holland%2C+D.%3B+Hussain%2C+S.&rft.au=Aitamurto%2C+T.%3B+Holland%2C+D.%3B+Hussain%2C+S.&rft.date=2015&rft.volume=31&rft.issue=4&rft.pages=17%E2%80%9329&rft_id=info:doi\/10.1162%2FDESI_a_00348&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuizinghOpen11-37\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HuizinghOpen11_37-0\">32.0<\/a><\/sup> <sup><a href=\"#cite_ref-HuizinghOpen11_37-1\">32.1<\/a><\/sup> <sup><a href=\"#cite_ref-HuizinghOpen11_37-2\">32.2<\/a><\/sup> <sup><a href=\"#cite_ref-HuizinghOpen11_37-3\">32.3<\/a><\/sup> <sup><a href=\"#cite_ref-HuizinghOpen11_37-4\">32.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Huizingh, E.K.R.E. (2011). \"Open innovation: State of the art and future perspectives\". <i>Technovation<\/i> <b>31<\/b> (1): 2\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.technovation.2010.10.002\" target=\"_blank\">10.1016\/j.technovation.2010.10.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+innovation%3A+State+of+the+art+and+future+perspectives&rft.jtitle=Technovation&rft.aulast=Huizingh%2C+E.K.R.E.&rft.au=Huizingh%2C+E.K.R.E.&rft.date=2011&rft.volume=31&rft.issue=1&rft.pages=2%E2%80%939&rft_id=info:doi\/10.1016%2Fj.technovation.2010.10.002&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BonvoisinCurrent17-38\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BonvoisinCurrent17_38-0\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-BonvoisinCurrent17_38-1\">33.1<\/a><\/sup> <sup><a href=\"#cite_ref-BonvoisinCurrent17_38-2\">33.2<\/a><\/sup> <sup><a href=\"#cite_ref-BonvoisinCurrent17_38-3\">33.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bonvoisin, J.; Laetitia, T.; Mies, R. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.designsociety.org\/publication\/39565\/Current+state+of+practices+in+open+source+product+development\" target=\"_blank\">\"Current state of practices in open source product development\"<\/a>. <i>Proceedings of the 21st International Conference on Engineering Design<\/i> <b>2<\/b>: 111\u2013120<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.designsociety.org\/publication\/39565\/Current+state+of+practices+in+open+source+product+development\" target=\"_blank\">https:\/\/www.designsociety.org\/publication\/39565\/Current+state+of+practices+in+open+source+product+development<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Current+state+of+practices+in+open+source+product+development&rft.jtitle=Proceedings+of+the+21st+International+Conference+on+Engineering+Design&rft.aulast=Bonvoisin%2C+J.%3B+Laetitia%2C+T.%3B+Mies%2C+R.+et+al.&rft.au=Bonvoisin%2C+J.%3B+Laetitia%2C+T.%3B+Mies%2C+R.+et+al.&rft.date=2017&rft.volume=2&rft.pages=111%E2%80%93120&rft_id=https%3A%2F%2Fwww.designsociety.org%2Fpublication%2F39565%2FCurrent%2Bstate%2Bof%2Bpractices%2Bin%2Bopen%2Bsource%2Bproduct%2Bdevelopment&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GacekTheMany04-39\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GacekTheMany04_39-0\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-GacekTheMany04_39-1\">34.1<\/a><\/sup> <sup><a href=\"#cite_ref-GacekTheMany04_39-2\">34.2<\/a><\/sup> <sup><a href=\"#cite_ref-GacekTheMany04_39-3\">34.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gacek, C.; Arief, B. (2004). \"The many meanings of open source\". <i>IEEE Software<\/i> <b>21<\/b> (1): 34\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FMS.2004.1259206\" target=\"_blank\">10.1109\/MS.2004.1259206<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+many+meanings+of+open+source&rft.jtitle=IEEE+Software&rft.aulast=Gacek%2C+C.%3B+Arief%2C+B.&rft.au=Gacek%2C+C.%3B+Arief%2C+B.&rft.date=2004&rft.volume=21&rft.issue=1&rft.pages=34%E2%80%9340&rft_id=info:doi\/10.1109%2FMS.2004.1259206&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BalkaOpen09-40\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BalkaOpen09_40-0\">35.0<\/a><\/sup> <sup><a href=\"#cite_ref-BalkaOpen09_40-1\">35.1<\/a><\/sup> <sup><a href=\"#cite_ref-BalkaOpen09_40-2\">35.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Balka, K.; Raasch, C.; Herstatt, C. (2009). \"Open source enters the world of atoms: A statistical analysis of open design\". <i>First Monday<\/i> <b>14<\/b> (11). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5210%2Ffm.v14i11.2670\" target=\"_blank\">10.5210\/fm.v14i11.2670<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+source+enters+the+world+of+atoms%3A+A+statistical+analysis+of+open+design&rft.jtitle=First+Monday&rft.aulast=Balka%2C+K.%3B+Raasch%2C+C.%3B+Herstatt%2C+C.&rft.au=Balka%2C+K.%3B+Raasch%2C+C.%3B+Herstatt%2C+C.&rft.date=2009&rft.volume=14&rft.issue=11&rft_id=info:doi\/10.5210%2Ffm.v14i11.2670&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GibbBuild14-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GibbBuild14_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Gibb, A. (2014). <i>Building Open Source Hardware: DIY Manufacturing for Hackers and Makers<\/i>. Addison-Wesley Professional. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780321906045.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Building+Open+Source+Hardware%3A+DIY+Manufacturing+for+Hackers+and+Makers&rft.aulast=Gibb%2C+A.&rft.au=Gibb%2C+A.&rft.date=2014&rft.pub=Addison-Wesley+Professional&rft.isbn=9780321906045&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JacobsProduct07-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JacobsProduct07_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jacobs, M.A. (2007). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ecommons.udayton.edu\/mis_fac_pub\/94\" target=\"_blank\">\"Product Complexity: A Definition and Impacts on Operations\"<\/a>. <i>Decision Line<\/i> <b>38<\/b> (5): 6\u20139, 21<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ecommons.udayton.edu\/mis_fac_pub\/94\" target=\"_blank\">https:\/\/ecommons.udayton.edu\/mis_fac_pub\/94<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Product+Complexity%3A+A+Definition+and+Impacts+on+Operations&rft.jtitle=Decision+Line&rft.aulast=Jacobs%2C+M.A.&rft.au=Jacobs%2C+M.A.&rft.date=2007&rft.volume=38&rft.issue=5&rft.pages=6%E2%80%939%2C+21&rft_id=https%3A%2F%2Fecommons.udayton.edu%2Fmis_fac_pub%2F94&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BonvoisinDesign17-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BonvoisinDesign17_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bonvoisin, J.; Galla, J.K.; Prendeville, S. (2017). \"Design Principles for Do-It-Yourself Production\". <i>Proceedings from Sustainable Design and Manufacturing 2017<\/i>: 77\u201386. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-57078-5_8\" target=\"_blank\">10.1007\/978-3-319-57078-5_8<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Design+Principles+for+Do-It-Yourself+Production&rft.jtitle=Proceedings+from+Sustainable+Design+and+Manufacturing+2017&rft.aulast=Bonvoisin%2C+J.%3B+Galla%2C+J.K.%3B+Prendeville%2C+S.&rft.au=Bonvoisin%2C+J.%3B+Galla%2C+J.K.%3B+Prendeville%2C+S.&rft.date=2017&rft.pages=77%E2%80%9386&rft_id=info:doi\/10.1007%2F978-3-319-57078-5_8&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Rodriguez-ToroProduct04-45\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Rodriguez-ToroProduct04_45-0\">39.0<\/a><\/sup> <sup><a href=\"#cite_ref-Rodriguez-ToroProduct04_45-1\">39.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rodriguez-Toro, C.A.; Jared, G.; Swift, K. (2004). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.designsociety.org\/publication\/19794\/PRODUCT-DEVELOPMENT+COMPLEXITY+METRICS%3A+A+FRAMEWORK+FOR+PROACTIVE-DFA+IMPLEMENTATION\" target=\"_blank\">\"Product-Development Complexity Metrics: A Framework for Proactive-DFA Implementation\"<\/a>. <i>Proceedings of DESIGN 2004<\/i>: 483\u201390<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.designsociety.org\/publication\/19794\/PRODUCT-DEVELOPMENT+COMPLEXITY+METRICS%3A+A+FRAMEWORK+FOR+PROACTIVE-DFA+IMPLEMENTATION\" target=\"_blank\">https:\/\/www.designsociety.org\/publication\/19794\/PRODUCT-DEVELOPMENT+COMPLEXITY+METRICS%3A+A+FRAMEWORK+FOR+PROACTIVE-DFA+IMPLEMENTATION<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Product-Development+Complexity+Metrics%3A+A+Framework+for+Proactive-DFA+Implementation&rft.jtitle=Proceedings+of+DESIGN+2004&rft.aulast=Rodriguez-Toro%2C+C.A.%3B+Jared%2C+G.%3B+Swift%2C+K.&rft.au=Rodriguez-Toro%2C+C.A.%3B+Jared%2C+G.%3B+Swift%2C+K.&rft.date=2004&rft.pages=483%E2%80%9390&rft_id=https%3A%2F%2Fwww.designsociety.org%2Fpublication%2F19794%2FPRODUCT-DEVELOPMENT%2BCOMPLEXITY%2BMETRICS%253A%2BA%2BFRAMEWORK%2BFOR%2BPROACTIVE-DFA%2BIMPLEMENTATION&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KyriakouKnowledge17-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KyriakouKnowledge17_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kyriakou, H.; Nickerson, J.V.; Sabnis, G. (2017). \"Knowledge Reuse for Customization: Metamodels in an Open Design Community for 3D Printing\". <i>MIS Quarterly<\/i> <b>41<\/b> (1): 315-332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.25300%2FMISQ%2F2017%2F41.1.17\" target=\"_blank\">10.25300\/MISQ\/2017\/41.1.17<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Knowledge+Reuse+for+Customization%3A+Metamodels+in+an+Open+Design+Community+for+3D+Printing&rft.jtitle=MIS+Quarterly&rft.aulast=Kyriakou%2C+H.%3B+Nickerson%2C+J.V.%3B+Sabnis%2C+G.&rft.au=Kyriakou%2C+H.%3B+Nickerson%2C+J.V.%3B+Sabnis%2C+G.&rft.date=2017&rft.volume=41&rft.issue=1&rft.pages=315-332&rft_id=info:doi\/10.25300%2FMISQ%2F2017%2F41.1.17&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EhlsDiversity15-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EhlsDiversity15_47-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ehls, D. (2015). \"Diversity of Participants in Open Source Projects: Comparing Individual Demographics and Participation Rationales in Software, Content, Fun, and Business Communities\". In Herstatt, C.; Ehls, D.. <i>Open Source Innovation: The Phenomenon, Participant's Behaviour, Business Implications<\/i>. Routledge. pp. 63\u201380. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781315754482.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Diversity+of+Participants+in+Open+Source+Projects%3A+Comparing+Individual+Demographics+and+Participation+Rationales+in+Software%2C+Content%2C+Fun%2C+and+Business+Communities&rft.atitle=Open+Source+Innovation%3A+The+Phenomenon%2C+Participant%27s+Behaviour%2C+Business+Implications&rft.aulast=Ehls%2C+D.&rft.au=Ehls%2C+D.&rft.date=2015&rft.pages=pp.%26nbsp%3B63%E2%80%9380&rft.pub=Routledge&rft.isbn=9781315754482&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UlrichProduct11-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UlrichProduct11_48-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ulrich, K.T.; Eppinger, S.D. (2011). <i>Product Design and Development<\/i> (5th ed.). McGraw-Hill Education. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780073404776.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Product+Design+and+Development&rft.aulast=Ulrich%2C+K.T.%3B+Eppinger%2C+S.D.&rft.au=Ulrich%2C+K.T.%3B+Eppinger%2C+S.D.&rft.date=2011&rft.edition=5th&rft.pub=McGraw-Hill+Education&rft.isbn=9780073404776&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSHABest13-49\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OSHABest13_49-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Open Source Hardware Association (2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.oshwa.org\/sharing-best-practices\/\" target=\"_blank\">\"Best Practices for Open-Source Hardware 1.0\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.oshwa.org\/sharing-best-practices\/\" target=\"_blank\">https:\/\/www.oshwa.org\/sharing-best-practices\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Best+Practices+for+Open-Source+Hardware+1.0&rft.atitle=&rft.aulast=Open+Source+Hardware+Association&rft.au=Open+Source+Hardware+Association&rft.date=2013&rft_id=https%3A%2F%2Fwww.oshwa.org%2Fsharing-best-practices%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChoiASurv10-50\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChoiASurv10_50-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Choi, S.-S.; Cha, S.-H.; Tappert, C.C. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.iiisci.org\/journal\/sci\/FullText.asp?var=&id=GS315JG\" target=\"_blank\">\"A Survey of Binary Similarity and Distance Measures\"<\/a>. <i>Journal of Systemics, Cybernetics and Informatics<\/i> <b>8<\/b> (1): 43\u201348<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.iiisci.org\/journal\/sci\/FullText.asp?var=&id=GS315JG\" target=\"_blank\">http:\/\/www.iiisci.org\/journal\/sci\/FullText.asp?var=&id=GS315JG<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Survey+of+Binary+Similarity+and+Distance+Measures&rft.jtitle=Journal+of+Systemics%2C+Cybernetics+and+Informatics&rft.aulast=Choi%2C+S.-S.%3B+Cha%2C+S.-H.%3B+Tappert%2C+C.C.&rft.au=Choi%2C+S.-S.%3B+Cha%2C+S.-H.%3B+Tappert%2C+C.C.&rft.date=2010&rft.volume=8&rft.issue=1&rft.pages=43%E2%80%9348&rft_id=http%3A%2F%2Fwww.iiisci.org%2Fjournal%2Fsci%2FFullText.asp%3Fvar%3D%26id%3DGS315JG&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TibshiraniEstim02-51\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TibshiraniEstim02_51-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tibshirani, R.; Walther, G.; Hastie, T. (2001). \"Estimating the number of clusters in a data set via the gap statistic\". <i>Statistical Methodology Series B<\/i> <b>63<\/b> (2): 411\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2F1467-9868.00293\" target=\"_blank\">10.1111\/1467-9868.00293<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Estimating+the+number+of+clusters+in+a+data+set+via+the+gap+statistic&rft.jtitle=Statistical+Methodology+Series+B&rft.aulast=Tibshirani%2C+R.%3B+Walther%2C+G.%3B+Hastie%2C+T.&rft.au=Tibshirani%2C+R.%3B+Walther%2C+G.%3B+Hastie%2C+T.&rft.date=2001&rft.volume=63&rft.issue=2&rft.pages=411%E2%80%9323&rft_id=info:doi\/10.1111%2F1467-9868.00293&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSHAReqs17-52\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OSHAReqs17_52-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Open Source Hardware Association (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/certificate.oshwa.org\/\" target=\"_blank\">\"Requirements for Certification\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/certificate.oshwa.org\/\" target=\"_blank\">https:\/\/certificate.oshwa.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Requirements+for+Certification&rft.atitle=&rft.aulast=Open+Source+Hardware+Association&rft.au=Open+Source+Hardware+Association&rft.date=2017&rft_id=https%3A%2F%2Fcertificate.oshwa.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BonvoisinOpenness17-53\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BonvoisinOpenness17_53-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bonvoisin, J.; Schmidt, K.C. (07 July 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/depositonce.tu-berlin.de\/handle\/11303\/6430\" target=\"_blank\">\"Openness assessment of 132 open source hardware products\"<\/a>. <i>DepositOnce<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.14279%2Fdepositonce-5977\" target=\"_blank\">10.14279\/depositonce-5977<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/depositonce.tu-berlin.de\/handle\/11303\/6430\" target=\"_blank\">https:\/\/depositonce.tu-berlin.de\/handle\/11303\/6430<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Openness+assessment+of+132+open+source+hardware+products&rft.atitle=DepositOnce&rft.aulast=Bonvoisin%2C+J.%3B+Schmidt%2C+K.C.&rft.au=Bonvoisin%2C+J.%3B+Schmidt%2C+K.C.&rft.date=07+July+2017&rft_id=info:doi\/10.14279%2Fdepositonce-5977&rft_id=https%3A%2F%2Fdepositonce.tu-berlin.de%2Fhandle%2F11303%2F6430&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_%22source%22_of_open-source_hardware%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance. To more easily differentiate footnotes from references, the original footnotes (which where numbered) were updated to use lowercase letters. Some footnotes simply referencing web pages were either turned into inline links or removed entirely for being superfluous. In the original, the citation for the Free Software Foundation 2015 is inadvertently missing; the presumed citation is added in this version.<\/blockquote>\n\n<!-- \nNewPP limit report\nCached time: 20200707204403\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.954 seconds\nReal time usage: 3.524 seconds\nPreprocessor visited node count: 34647\/1000000\nPreprocessor generated node count: 42266\/1000000\nPost\u2010expand include size: 222152\/2097152 bytes\nTemplate argument size: 80785\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 727.429 1 - -total\n 86.44% 628.810 2 - Template:Reflist\n 73.63% 535.641 47 - Template:Citation\/core\n 51.94% 377.804 32 - Template:Cite_journal\n 13.14% 95.574 7 - Template:Cite_book\n 12.03% 87.540 8 - Template:Cite_web\n 5.10% 37.114 1 - Template:Infobox_journal_article\n 4.86% 35.382 1 - Template:Infobox\n 4.15% 30.203 34 - Template:Citation\/identifier\n 3.60% 26.158 54 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11381-0!*!0!!en!5!*!math=5 and timestamp 20200707204400 and revision id 37471\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_%22source%22_of_open-source_hardware%3F\">https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_%22source%22_of_open-source_hardware%3F<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","3ab9770e7fca9810f5573456215e3853_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/50\/Fig1_Bonvoisin_JOfOpenHard2017_1-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0b\/Fig2_Bonvoisin_JOfOpenHard2017_1-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/e1\/Fig3_Bonvoisin_JOfOpenHard2017_1-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/e5\/Fig4_Bonvoisin_JOfOpenHard2017_1-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/fd\/Fig5_Bonvoisin_JOfOpenHard2017_1-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/e0\/Fig6_Bonvoisin_JOfOpenHard2017_1-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/37\/Fig7_Bonvoisin_JOfOpenHard2017_1-1.png"],"3ab9770e7fca9810f5573456215e3853_timestamp":1594154640,"c3c7e4673674f7b161f8fb6199af909a_type":"article","c3c7e4673674f7b161f8fb6199af909a_title":"Point-of-care RNA-based diagnostic device for COVID-19 (Yang et al. 2020)","c3c7e4673674f7b161f8fb6199af909a_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19","c3c7e4673674f7b161f8fb6199af909a_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Point-of-care RNA-based diagnostic device for COVID-19\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nPoint-of-care RNA-based diagnostic device for COVID-19Journal\n \nDiagnosticsAuthor(s)\n \nYang, Ting; Wang, Yung-Chih; Shen, Ching-Fen; Cheng, Chao-MinAuthor affiliation(s)\n \nNational Tsing Hua University, National Defense Medical Center, National Cheng Kung UniversityPrimary contact\n \nEmail: chaomin at mx dot nthu dot edu dot twYear published\n \n2020Volume and issue\n \n10(3)Article #\n \n165DOI\n \n10.3390\/diagnostics10030165ISSN\n \n2075-4418Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/2075-4418\/10\/3\/165\/htmDownload\n \nhttps:\/\/www.mdpi.com\/2075-4418\/10\/3\/165\/pdf (PDF)\n\nContents\n\n1 Point-of-care RNA-based diagnostic device for COVID-19 \n2 Acknowledgements \n\n2.1 Funding \n2.2 Conflicts of interest \n\n\n3 References \n4 Notes \n\n\n\nPoint-of-care RNA-based diagnostic device for COVID-19 \nAt the end of 2019, the novel coronavirus disease 2019 (COVID-19), a fast-spreading respiratory disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was reported in Wuhan, China and has now affected over 123 countries globally. As of March 14, 2020, the death toll has exceeded 5,400, and there have been 145,000 confirmed cases, causing not only a huge medical health burden but also tremendous economic losses worldwide.[1] Unlike previous coronaviruses that caused large-scale epidemics, such as the Middle East respiratory syndrome (MERS) and severe acute respiratory syndrome (SARS), the transmission rate for COVID-19 is much higher, with an average of two to three people becoming infected for every already infected person.[2] Currently, there are two primary methods for diagnosing COVID-19: (1) a lateral flow immunoassay, which is a common point-of-care (POC) diagnostic approach that detects antibodies against specific viruses (e.g., SARS-CoV-2) in patient samples; and (2) a molecular-based assay. The current standard approach for screening COVID-19 requires a reverse real-time polymerase chain reaction (rRT-PCR) assay, which can be carried out using a variety of clinical specimens, including bronchoalveolar lavage fluid, fibrobronchoscope brush biopsies, sputum, nasal swabs, nasopharyngeal swabs, feces, or blood.[3] This approach relies on expensive facilities, well-trained staff, and is often time-consuming, leaving a rapidly rising number of potential cases untested and opening a gaping hole in disease prevention efforts. Moreover, traveling to a clinical setting for testing increases the risk of spreading the disease and adds strain to a resource-limited healthcare system. For these reasons, an alternative, rapid, inexpensive, easy-to-use, and sensitive COVID-19 diagnostic tool must be developed for use by nonclinical individuals in their homes.\nA recent study conducted by El-Tholoth et al. at the University of Pennsylvania in mid-February of 2020 describes a novel closed-tube COVID-19 assay that includes pathogen nuclei acid amplification and detection.[4] Because viral samples are small and standard practices require RT-PCR, false-negative test results are possible and patients may consequently become more seriously ill.[5] To improve COVID-19 diagnostic test sensitivity, El-Tholoth\u2019s group developed a closed-tube Penn-RAMP, a two-stage isothermal dsDNA amplification method that utilized both recombinase polymerase amplification (RPA) and loop-mediated isothermal amplification (LAMP) techniques in a single tube. To perform the LAMP test specifically for COVID-19, El-Tholoth et al. selected conserved COVID-19 sequences using Clustal X and further designed LAMP primers. In order to make the detection process simpler, they used leucocrystal violet (LCV) dye as a chromogenic reagent, providing an obvious, deep violet color change that could be observed with the naked eye. The entire diagnostic process was relatively simple and required only a single tube for reaction. In this process, the RPA mixture was loaded onto the inside of the tube lid and the LAMP mixture (ratio of 1:9) was placed within the tube itself. The tube was subsequently sealed and incubated at 38 degrees Celsius for 15 to 20 minutes in a thermal cycler to facilitate the RPA reaction. The tube was then inverted several times and incubated at 63 degrees Celsius for 40 minutes. Because they did not have access to actual COVID-19 samples, El-Tholoth et al. used samples of healthy nasal mucosa spiked with inactivated HIV particles and other pathogens as their test materials. The Penn-RAMP process provided greater sensitivity than RT-PCR or LAMP alone. When using limited viral load, Penn-RAMP provided 100 times better sensitivity than a single LAMP test. Compared to a LAMP assay, which requires sophisticated equipment and must be run at a fixed temperature, the Penn-RAMP process requires less energy cost, is easier to execute, and can be completed in clinical or home settings.\nThe abovementioned research describes essential materials and protocol for a single-tube COVID-19 test that can be completed without visiting a clinical setting. Because standard clinical RT-PCR testing is unlikely to meet rising test demand, there is a critical need for the development of alternative approaches for home-based point-of-care (POC) testing. These approaches may include LAMP assay and lateral flow assay techniques. Here, we describe a potential RNA-based POC diagnostic device for detecting COVID-19 that combines both a paper-based POC diagnostic device and LAMP assay technology. This convenient device can be integrated with a smartphone application to provide a rapid, sensitive, and highly accessible COVID-19 diagnostic tool (Figure 1).\n\n\n\n\n\n\n\n\n\n Figure 1. Schematic of the point-of-care (POC) RNA-based novel coronavirus disease (COVID-19) diagnostic device. In this figure, the conceptual workflow of integrating both point-of-care diagnostic and loop-mediated isothermal amplification (LAMP) assay with mobile device was presented. People quarantined at home can collect their infectious specimen through nasal swab; after adding specific reagents for LAMP reaction, the colorimetric result can be observed on paper. By recording the colorimetric change through a mobile phone camera, the user could upload the result to cloud storage by internet. Clinicians or health-care specialist could thus analyze the result and offer immediate information for both patient and the government. Since the whole diagnostic process can be performed at home, the suspected case does not need to travel to hospital and can in turn lower the risk of spreading the disease.\n\n\n\nThe concept for this tool is derived from previous research on paper-based nucleic acid detection employing RT-LAMP assay amplification.[6] Paper-based diagnostic tools have been widely applied for a variety of biochemical assays due to their low cost, ease of use, and speed. They have been employed to test a range of sample sources, including blood, urine, tears, and vaginal fluid, and could be easily adapted to accept nasal swab samples for viral detection. We previously developed a paper-based platform for fluorescent nucleic acid screening using nucleic ladders. This research provided considerable insight into fluorescent probe selection and viral detection and required only a small sample volume. The potential rapid and easy-to-use paper-based LAMP assay for COVID-19 could be used in combination with a smartphone application to facilitate test results recording\/sharing. Using this tool, a home-quarantined individual could easily self-collect a nasal swab sample; perform a LAMP assay; and observe a visible, colorimetric test result that could then be recorded and shared with clinicians or healthcare professionals via the internet.\nAs the number of confirmed cases of COVID-19 rises rapidly throughout the world, more and more patients will require reliable testing. A rapid, inexpensive, and easy-to-use POC diagnostic device integrated with a smartphone could reduce transportation needs, lower the risk of spreading infection, alleviate the strain on the healthcare system, and mitigate the cost of care for both individuals and the government. We believe that research into the development of a paper-based RNA assay for use in combination with a smartphone application can provide new insights into designing POC COVID-19 diagnostics and ultimately improve the health care system to combat this and similar diseases.\n\nAcknowledgements \nFunding \nThis research received no external funding.\n\nConflicts of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 \"Coronavirus COVID-19 outbreak: Latest news, information and updates\". Pharmaceutical Technology. Verdict Media Limited. March 2020. https:\/\/www.pharmaceutical-technology.com\/special-focus\/covid-19\/coronavirus-covid-19-outbreak-latest-information-news-and-updates\/ . Retrieved 14 March 2020 .   \n\n\u2191 Gates, B. (2020). \"Responding to Covid-19 - A Once-in-a-Century Pandemic?\". New England Journal of Medicine. doi:10.1056\/NEJMp2003762. PMID 32109012.   \n\n\u2191 Wang, W.; Xu, Y.; Gao, R. et al. (2020). \"Detection of SARS-CoV-2 in Different Types of Clinical Specimens\". JAMA. doi:10.1001\/jama.2020.3786. PMC PMC7066521. PMID 32159775. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7066521 .   \n\n\u2191 El-Tholoth, M.; Bau, H.H.; Song, J. (2020). \"A Single and Two-Stage, Closed-Tube, Molecular Test for the 2019 Novel Coronavirus (COVID-19) at Home, Clinic, and Points of Entry\". ChemRxiv. doi:10.26434\/chemrxiv.11860137.v1.   \n\n\u2191 Xie, X.; Zhong, Z.; Zhao, W. et al. (2020). \"Chest CT for Typical 2019-nCoV Pneumonia: Relationship to Negative RT-PCR Testing\". Radiology: 200343. doi:10.1148\/radiol.2020200343. PMID 32049601.   \n\n\u2191 Lo, S.J.; Yang, S.C.; Yao, D.J. et al. (2013). \"Molecular-level dengue fever diagnostic devices made out of paper\". Lab on a Chip 13 (14): 2686\u201392. doi:10.1039\/c3lc50135c. PMID 23563693.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19\">https:\/\/www.limswiki.org\/index.php\/Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on clinical researchLIMSwiki journal articles on epidemiology\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 30 March 2020, at 18:50.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 161 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","c3c7e4673674f7b161f8fb6199af909a_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Point-of-care_RNA-based_diagnostic_device_for_COVID-19 skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Point-of-care RNA-based diagnostic device for COVID-19<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Point-of-care_RNA-based_diagnostic_device_for_COVID-19\">Point-of-care RNA-based diagnostic device for COVID-19<\/span><\/h2>\n<p>At the end of 2019, the novel <a href=\"https:\/\/www.limswiki.org\/index.php\/Coronavirus_disease_2019\" title=\"Coronavirus disease 2019\" class=\"wiki-link\" data-key=\"68331dff29df205bcb39c3ad9599c30c\">coronavirus disease 2019<\/a> (COVID-19), a fast-spreading respiratory disease caused by <a href=\"https:\/\/www.limswiki.org\/index.php\/Severe_acute_respiratory_syndrome_coronavirus_2\" title=\"Severe acute respiratory syndrome coronavirus 2\" class=\"wiki-link\" data-key=\"beddd8bfa6022d0f538d26cdefb7df5c\">severe acute respiratory syndrome coronavirus 2<\/a> (SARS-CoV-2), was reported in Wuhan, China and has now affected over 123 countries globally. As of March 14, 2020, the death toll has exceeded 5,400, and there have been 145,000 confirmed cases, causing not only a huge medical health burden but also tremendous economic losses worldwide.<sup id=\"rdp-ebb-cite_ref-PTCorona20_1-0\" class=\"reference\"><a href=\"#cite_note-PTCorona20-1\">[1]<\/a><\/sup> Unlike previous <a href=\"https:\/\/www.limswiki.org\/index.php\/Coronavirus\" title=\"Coronavirus\" class=\"wiki-link\" data-key=\"86c887aaa85c1b2b96fd478c10703204\">coronaviruses<\/a> that caused large-scale , such as the <a href=\"https:\/\/www.limswiki.org\/index.php\/Middle_East_respiratory_syndrome\" title=\"Middle East respiratory syndrome\" class=\"wiki-link\" data-key=\"6a290adf3ac17e4b8a75ef1ce0b28afd\">Middle East respiratory syndrome<\/a> (MERS) and <a href=\"https:\/\/www.limswiki.org\/index.php\/Severe_acute_respiratory_syndrome\" title=\"Severe acute respiratory syndrome\" class=\"wiki-link\" data-key=\"11abe2043ece64ad43ee0052402c5cec\">severe acute respiratory syndrome<\/a> (SARS), the transmission rate for COVID-19 is much higher, with an average of two to three people becoming infected for every already infected person.<sup id=\"rdp-ebb-cite_ref-GatesRespond20_2-0\" class=\"reference\"><a href=\"#cite_note-GatesRespond20-2\">[2]<\/a><\/sup> Currently, there are two primary methods for diagnosing COVID-19: (1) a lateral flow <a href=\"https:\/\/www.limswiki.org\/index.php\/Immunoassay\" title=\"Immunoassay\" class=\"wiki-link\" data-key=\"ce415f577070e6fa6afa0a305f5f0247\">immunoassay<\/a>, which is a common point-of-care (POC) diagnostic approach that detects antibodies against specific viruses (e.g., SARS-CoV-2) in patient samples; and (2) a <a href=\"https:\/\/www.limswiki.org\/index.php\/Molecular_diagnostics\" title=\"Molecular diagnostics\" class=\"wiki-link\" data-key=\"8fc14cae7a6fbac9a53fae1394fae7ee\">molecular-based assay<\/a>. The current standard approach for screening COVID-19 requires a reverse real-time <a href=\"https:\/\/www.limswiki.org\/index.php\/Polymerase_chain_reaction\" title=\"Polymerase chain reaction\" class=\"wiki-link\" data-key=\"f6569fb01ef396379f9f4efa4527e715\">polymerase chain reaction<\/a> (rRT-PCR) assay, which can be carried out using a variety of clinical specimens, including <a href=\"https:\/\/www.limswiki.org\/index.php\/Bronchoalveolar_lavage\" title=\"Bronchoalveolar lavage\" class=\"wiki-link\" data-key=\"500aa79d30c49b81ed6c58bc319231dc\">bronchoalveolar lavage<\/a> fluid, fibrobronchoscope brush biopsies, sputum, nasal swabs, <a href=\"https:\/\/www.limswiki.org\/index.php\/Nasopharyngeal_swab\" title=\"Nasopharyngeal swab\" class=\"wiki-link\" data-key=\"18d5d4e09d1fc5ddb05a22b36ace9daf\">nasopharyngeal swabs<\/a>, feces, or blood.<sup id=\"rdp-ebb-cite_ref-WangDetect20_3-0\" class=\"reference\"><a href=\"#cite_note-WangDetect20-3\">[3]<\/a><\/sup> This approach relies on expensive facilities, well-trained staff, and is often time-consuming, leaving a rapidly rising number of potential cases untested and opening a gaping hole in disease prevention efforts. Moreover, traveling to a clinical setting for testing increases the risk of spreading the disease and adds strain to a resource-limited healthcare system. For these reasons, an alternative, rapid, inexpensive, easy-to-use, and sensitive COVID-19 diagnostic tool must be developed for use by nonclinical individuals in their homes.\n<\/p><p>A recent study conducted by El-Tholoth <i>et al.<\/i> at the University of Pennsylvania in mid-February of 2020 describes a novel closed-tube COVID-19 assay that includes pathogen <a href=\"https:\/\/www.limswiki.org\/index.php\/Nucleic_acid_test\" title=\"Nucleic acid test\" class=\"wiki-link\" data-key=\"9b850e87cf2257909a3cde1f7ea4dd94\">nuclei acid amplification and detection<\/a>.<sup id=\"rdp-ebb-cite_ref-El-TholothASingle20_4-0\" class=\"reference\"><a href=\"#cite_note-El-TholothASingle20-4\">[4]<\/a><\/sup> Because viral samples are small and standard practices require RT-PCR, false-negative test results are possible and patients may consequently become more seriously ill.<sup id=\"rdp-ebb-cite_ref-XieChest20_5-0\" class=\"reference\"><a href=\"#cite_note-XieChest20-5\">[5]<\/a><\/sup> To improve COVID-19 diagnostic test sensitivity, El-Tholoth\u2019s group developed a closed-tube Penn-RAMP, a two-stage isothermal dsDNA amplification method that utilized both <a href=\"https:\/\/www.limswiki.org\/index.php\/Recombinase_polymerase_amplification\" title=\"Recombinase polymerase amplification\" class=\"wiki-link\" data-key=\"716ff4ec91f1ddf64cc1458900b6a678\">recombinase polymerase amplification<\/a> (RPA) and <a href=\"https:\/\/www.limswiki.org\/index.php\/Loop-mediated_isothermal_amplification\" title=\"Loop-mediated isothermal amplification\" class=\"wiki-link\" data-key=\"e71e4c1cfffeaf6781dd13b0ac1cc2a9\">loop-mediated isothermal amplification<\/a> (LAMP) techniques in a single tube. To perform the LAMP test specifically for COVID-19, El-Tholoth <i>et al.<\/i> selected conserved COVID-19 sequences using Clustal X and further designed LAMP primers. In order to make the detection process simpler, they used leucocrystal violet (LCV) dye as a chromogenic reagent, providing an obvious, deep violet color change that could be observed with the naked eye. The entire diagnostic process was relatively simple and required only a single tube for reaction. In this process, the RPA mixture was loaded onto the inside of the tube lid and the LAMP mixture (ratio of 1:9) was placed within the tube itself. The tube was subsequently sealed and incubated at 38 degrees Celsius for 15 to 20 minutes in a thermal cycler to facilitate the RPA reaction. The tube was then inverted several times and incubated at 63 degrees Celsius for 40 minutes. Because they did not have access to actual COVID-19 samples, El-Tholoth <i>et al.<\/i> used samples of healthy nasal mucosa spiked with inactivated HIV particles and other pathogens as their test materials. The Penn-RAMP process provided greater sensitivity than RT-PCR or LAMP alone. When using limited viral load, Penn-RAMP provided 100 times better sensitivity than a single LAMP test. Compared to a LAMP assay, which requires sophisticated equipment and must be run at a fixed temperature, the Penn-RAMP process requires less energy cost, is easier to execute, and can be completed in clinical or home settings.\n<\/p><p>The abovementioned research describes essential materials and protocol for a single-tube COVID-19 test that can be completed without visiting a clinical setting. Because standard clinical RT-PCR testing is unlikely to meet rising test demand, there is a critical need for the development of alternative approaches for home-based point-of-care (POC) testing. These approaches may include LAMP assay and lateral flow assay techniques. Here, we describe a potential RNA-based POC diagnostic device for detecting COVID-19 that combines both a paper-based POC diagnostic device and LAMP assay technology. This convenient device can be integrated with a smartphone application to provide a rapid, sensitive, and highly accessible COVID-19 diagnostic tool (Figure 1).\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Yang_Diagnostics2020_10-3.png\" class=\"image wiki-link\" data-key=\"20ca0dea3e40c4a74f3a671e901b2108\"><img alt=\"Fig1 Yang Diagnostics2020 10-3.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3f\/Fig1_Yang_Diagnostics2020_10-3.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Schematic of the point-of-care (POC) RNA-based novel coronavirus disease (COVID-19) diagnostic device. In this figure, the conceptual workflow of integrating both point-of-care diagnostic and loop-mediated isothermal amplification (LAMP) assay with mobile device was presented. People quarantined at home can collect their infectious specimen through nasal swab; after adding specific reagents for LAMP reaction, the colorimetric result can be observed on paper. By recording the colorimetric change through a mobile phone camera, the user could upload the result to cloud storage by internet. Clinicians or health-care specialist could thus analyze the result and offer immediate information for both patient and the government. Since the whole diagnostic process can be performed at home, the suspected case does not need to travel to hospital and can in turn lower the risk of spreading the disease.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The concept for this tool is derived from previous research on paper-based nucleic acid detection employing RT-LAMP assay amplification.<sup id=\"rdp-ebb-cite_ref-LoMolec13_6-0\" class=\"reference\"><a href=\"#cite_note-LoMolec13-6\">[6]<\/a><\/sup> Paper-based diagnostic tools have been widely applied for a variety of biochemical assays due to their low cost, ease of use, and speed. They have been employed to test a range of sample sources, including blood, urine, tears, and vaginal fluid, and could be easily adapted to accept nasal swab samples for viral detection. We previously developed a paper-based platform for fluorescent nucleic acid screening using nucleic ladders. This research provided considerable insight into fluorescent probe selection and viral detection and required only a small sample volume. The potential rapid and easy-to-use paper-based LAMP assay for COVID-19 could be used in combination with a smartphone application to facilitate test results recording\/sharing. Using this tool, a home-quarantined individual could easily self-collect a nasal swab sample; perform a LAMP assay; and observe a visible, colorimetric test result that could then be recorded and shared with clinicians or healthcare professionals via the internet.\n<\/p><p>As the number of confirmed cases of COVID-19 rises rapidly throughout the world, more and more patients will require reliable testing. A rapid, inexpensive, and easy-to-use POC diagnostic device integrated with a smartphone could reduce transportation needs, lower the risk of spreading infection, alleviate the strain on the healthcare system, and mitigate the cost of care for both individuals and the government. We believe that research into the development of a paper-based RNA assay for use in combination with a smartphone application can provide new insights into designing POC COVID-19 diagnostics and ultimately improve the health care system to combat this and similar diseases.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research received no external funding.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-PTCorona20-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PTCorona20_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.pharmaceutical-technology.com\/special-focus\/covid-19\/coronavirus-covid-19-outbreak-latest-information-news-and-updates\/\" target=\"_blank\">\"Coronavirus COVID-19 outbreak: Latest news, information and updates\"<\/a>. <i>Pharmaceutical Technology<\/i>. Verdict Media Limited. March 2020<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.pharmaceutical-technology.com\/special-focus\/covid-19\/coronavirus-covid-19-outbreak-latest-information-news-and-updates\/\" target=\"_blank\">https:\/\/www.pharmaceutical-technology.com\/special-focus\/covid-19\/coronavirus-covid-19-outbreak-latest-information-news-and-updates\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 14 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Coronavirus+COVID-19+outbreak%3A+Latest+news%2C+information+and+updates&rft.atitle=Pharmaceutical+Technology&rft.date=March+2020&rft.pub=Verdict+Media+Limited&rft_id=https%3A%2F%2Fwww.pharmaceutical-technology.com%2Fspecial-focus%2Fcovid-19%2Fcoronavirus-covid-19-outbreak-latest-information-news-and-updates%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GatesRespond20-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GatesRespond20_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gates, B. (2020). \"Responding to Covid-19 - A Once-in-a-Century Pandemic?\". <i>New England Journal of Medicine<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMp2003762\" target=\"_blank\">10.1056\/NEJMp2003762<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32109012\" target=\"_blank\">32109012<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Responding+to+Covid-19+-+A+Once-in-a-Century+Pandemic%3F&rft.jtitle=New+England+Journal+of+Medicine&rft.aulast=Gates%2C+B.&rft.au=Gates%2C+B.&rft.date=2020&rft_id=info:doi\/10.1056%2FNEJMp2003762&rft_id=info:pmid\/32109012&rfr_id=info:sid\/en.wikipedia.org:Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangDetect20-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangDetect20_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, W.; Xu, Y.; Gao, R. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7066521\" target=\"_blank\">\"Detection of SARS-CoV-2 in Different Types of Clinical Specimens\"<\/a>. <i>JAMA<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2020.3786\" target=\"_blank\">10.1001\/jama.2020.3786<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7066521\/\" target=\"_blank\">PMC7066521<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32159775\" target=\"_blank\">32159775<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7066521\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7066521<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Detection+of+SARS-CoV-2+in+Different+Types+of+Clinical+Specimens&rft.jtitle=JAMA&rft.aulast=Wang%2C+W.%3B+Xu%2C+Y.%3B+Gao%2C+R.+et+al.&rft.au=Wang%2C+W.%3B+Xu%2C+Y.%3B+Gao%2C+R.+et+al.&rft.date=2020&rft_id=info:doi\/10.1001%2Fjama.2020.3786&rft_id=info:pmc\/PMC7066521&rft_id=info:pmid\/32159775&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7066521&rfr_id=info:sid\/en.wikipedia.org:Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-El-TholothASingle20-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-El-TholothASingle20_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">El-Tholoth, M.; Bau, H.H.; Song, J. (2020). \"A Single and Two-Stage, Closed-Tube, Molecular Test for the 2019 Novel Coronavirus (COVID-19) at Home, Clinic, and Points of Entry\". <i>ChemRxiv<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.26434%2Fchemrxiv.11860137.v1\" target=\"_blank\">10.26434\/chemrxiv.11860137.v1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Single+and+Two-Stage%2C+Closed-Tube%2C+Molecular+Test+for+the+2019+Novel+Coronavirus+%28COVID-19%29+at+Home%2C+Clinic%2C+and+Points+of+Entry&rft.jtitle=ChemRxiv&rft.aulast=El-Tholoth%2C+M.%3B+Bau%2C+H.H.%3B+Song%2C+J.&rft.au=El-Tholoth%2C+M.%3B+Bau%2C+H.H.%3B+Song%2C+J.&rft.date=2020&rft_id=info:doi\/10.26434%2Fchemrxiv.11860137.v1&rfr_id=info:sid\/en.wikipedia.org:Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XieChest20-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-XieChest20_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xie, X.; Zhong, Z.; Zhao, W. et al. (2020). \"Chest CT for Typical 2019-nCoV Pneumonia: Relationship to Negative RT-PCR Testing\". <i>Radiology<\/i>: 200343. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1148%2Fradiol.2020200343\" target=\"_blank\">10.1148\/radiol.2020200343<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32049601\" target=\"_blank\">32049601<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Chest+CT+for+Typical+2019-nCoV+Pneumonia%3A+Relationship+to+Negative+RT-PCR+Testing&rft.jtitle=Radiology&rft.aulast=Xie%2C+X.%3B+Zhong%2C+Z.%3B+Zhao%2C+W.+et+al.&rft.au=Xie%2C+X.%3B+Zhong%2C+Z.%3B+Zhao%2C+W.+et+al.&rft.date=2020&rft.pages=200343&rft_id=info:doi\/10.1148%2Fradiol.2020200343&rft_id=info:pmid\/32049601&rfr_id=info:sid\/en.wikipedia.org:Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LoMolec13-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LoMolec13_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lo, S.J.; Yang, S.C.; Yao, D.J. et al. (2013). \"Molecular-level dengue fever diagnostic devices made out of paper\". <i>Lab on a Chip<\/i> <b>13<\/b> (14): 2686\u201392. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1039%2Fc3lc50135c\" target=\"_blank\">10.1039\/c3lc50135c<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23563693\" target=\"_blank\">23563693<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Molecular-level+dengue+fever+diagnostic+devices+made+out+of+paper&rft.jtitle=Lab+on+a+Chip&rft.aulast=Lo%2C+S.J.%3B+Yang%2C+S.C.%3B+Yao%2C+D.J.+et+al.&rft.au=Lo%2C+S.J.%3B+Yang%2C+S.C.%3B+Yao%2C+D.J.+et+al.&rft.date=2013&rft.volume=13&rft.issue=14&rft.pages=2686%E2%80%9392&rft_id=info:doi\/10.1039%2Fc3lc50135c&rft_id=info:pmid\/23563693&rfr_id=info:sid\/en.wikipedia.org:Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204400\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.317 seconds\nReal time usage: 1.784 seconds\nPreprocessor visited node count: 6382\/1000000\nPreprocessor generated node count: 29238\/1000000\nPost\u2010expand include size: 43883\/2097152 bytes\nTemplate argument size: 14210\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 147.274 1 - -total\n 66.71% 98.242 1 - Template:Reflist\n 53.75% 79.165 6 - Template:Citation\/core\n 46.56% 68.577 5 - Template:Cite_journal\n 26.30% 38.735 1 - Template:Infobox_journal_article\n 25.04% 36.882 1 - Template:Infobox\n 16.66% 24.536 80 - Template:Infobox\/row\n 11.59% 17.070 1 - Template:Cite_web\n 8.51% 12.533 10 - Template:Citation\/identifier\n 3.82% 5.630 7 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11862-0!*!0!!en!5!* and timestamp 20200707204358 and revision id 38810\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19\">https:\/\/www.limswiki.org\/index.php\/Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","c3c7e4673674f7b161f8fb6199af909a_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3f\/Fig1_Yang_Diagnostics2020_10-3.png"],"c3c7e4673674f7b161f8fb6199af909a_timestamp":1594154638,"3eda7d7fb752e96c7816c5992924922e_type":"article","3eda7d7fb752e96c7816c5992924922e_title":"Ultra-high-performance liquid chromatography coupled with quadrupole-Orbitrap high-resolution mass spectrometry for multi-residue analysis of mycotoxins and pesticides in botanical nutraceuticals (Narv\u00e1ez et al. 2020)","3eda7d7fb752e96c7816c5992924922e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals","3eda7d7fb752e96c7816c5992924922e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Ultra-high-performance liquid chromatography coupled with quadrupole-Orbitrap high-resolution mass spectrometry for multi-residue analysis of mycotoxins and pesticides in botanical nutraceuticals\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nUltra-high-performance liquid chromatography coupled with quadrupole-Orbitrap high-resolution\r\nmass spectrometry for multi-residue analysis of mycotoxins and pesticides in botanical nutraceuticalsJournal\n \nToxinsAuthor(s)\n \nNarv\u00e1ez, Alfonso; Rodr\u00edguez-Carrasco, Yelko; Castaldo, Luigi; Izzo, Luana; Ritieni, AlbertoAuthor affiliation(s)\n \nUniversity of Naples \u201cFederico II\u201d, University of ValenciaPrimary contact\n \nEmail: yelko dot rodriguez at uv dot esYear published\n \n2020Volume and issue\n \n12(2)Page(s)\n \n114DOI\n \n10.3390\/toxins12020114ISSN\n \n2072-6651Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/2072-6651\/12\/2\/114\/htmDownload\n \nhttps:\/\/www.mdpi.com\/2072-6651\/12\/2\/114\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Results \n\n3.1 Optimization of extraction procedure \n\n3.1.1 Evaluation of the volume of extraction solvent \n3.1.2 Evaluation of the type of sorbent for clean-up \n\n\n3.2 Analytical method validation \n3.3 Application to commercial CBD-based products \n3.4 Identification of non-target compounds through retrospective analysis in studied samples \n\n\n4 Conclusions \n5 Materials and methods \n\n5.1 Chemicals and reagents \n5.2 Sampling \n5.3 Sample preparation \n5.4 UHPLC-Q-Orbitrap HRMS analysis \n5.5 Validation parameters \n5.6 Statistical analysis \n\n\n6 Acknowledgements \n\n6.1 Author contributions \n6.2 Funding \n6.3 Conflicts of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nCannabidiol (CBD) food supplements made of Cannabis sativa L. extracts have quickly become popular products due to their health-promoting effects. However, potential contaminants, such as mycotoxins and pesticides, can be coextracted during the manufacturing process and placed into the final product. Accordingly, a novel methodology using ultra-high-performance liquid chromatography coupled with quadrupole-Orbitrap high-resolution mass spectrometry (UHPLC-Q-Orbitrap HRMS) was developed to quantify 16 mycotoxins produced by major C. sativa fungi, followed by a post-target screening of 283 pesticides based on a comprehensive spectral library. The validated procedure was applied to 10 CBD-based products. Up to six different Fusarium mycotoxins were found in seven samples, the most prevalent being zearalenone (60%) and enniatin B1 (30%), both found at a maximum level of 11.6 ng\/g. Co-occurrence was observed in four samples, including one with enniatin B1, enniatin A, and enniatin A1. On the other hand, 46 different pesticides were detected after retrospective analysis. Ethoxyquin (50%), piperonyl butoxide (40%), simazine (30%), and cyanazine (30%) were the major residues found. These results highlight the necessity of monitoring contaminants in food supplements in order to ensure safe consumption, even more considering the increasing trend in their use. Furthermore, the developed procedure is proposed as a powerful analytical tool to evaluate the potential mycotoxin profile of these particular products. To our knowledge, this represents the first multi-class analysis of CBD-based supplements regarding mycotoxins and pesticide residues using high-resolution mass spectrometry techniques.\nKeywords: mycotoxins, pesticides, Q-Exactive Orbitrap, CBD capsule, nutraceutical\n\nIntroduction \nNutrition is known to be an essential component of quality state of health, and having an unbalanced diet can lead to several disorders and diseases.[1] Due to current lifestyles, new, convenient ways to maintain proper dietary habits are required. Nutraceuticals have emerged as an alternative to increase the input of nutrients, contributing to an improvement in health. These products are bioactive compounds naturally occurring in food or produced de novo in human metabolism, biologicals, or botanicals, each intended to impart a physiological or medicinal effect after ingestion.[2] They can be delivered either in foods and beverages or in other non-conventional forms, such as capsules, tablets, powders, or liquid extracts. In terms of marketing, nutraceuticals include a large number of different products packaged for specific groups by age, gender, physical condition, and activity level. The global market was valued at U.S. $109 billion in 2015 and was projected to reach U.S. $180 billion by 2020.[3]\nInside the variety of products classified as nutraceuticals, food supplements based on botanical ingredients represent the second largest segment, behind vitamins and minerals. Most recently, cannabidiol (CBD) dietary supplements made of Cannabis sativa L. extracts have quickly become popular products. CBD is a phytocannabinoid present in the resin secreted from trichomes in female C. sativa plants and is mainly found in inflorescences. The bioactivity of this compound has been related to an enhancement of its antioxidant and neurological activity, among others, by the promotion of several metabolic pathways.[4][5][6] However, the European Union (E.U.) does not consider CBD supplements as a novel food[7] and lets member states set their own rules over its marketing, leading to a convoluted situation in terms of regulation. Despite several ambiguities in its legislation, the European market for CBD-based supplements was valued at U.S. $318 million in 2018, with a strong growth projection.[8]\nDue to the complex nature of C. sativa and other botanicals, potential contaminants can be coextracted during the different stages of the manufacturing process and placed into the final product. Among all the potential non-desirable compounds in herbal-based supplements, mycotoxins and pesticides are the most commonly reported.[9][10] Mycotoxins are secondary metabolites mainly produced by the fungi genera Fusarium, Aspergillus, Penicillium, Claviceps, and Alternaria. These compounds can be present in food and feed commodities and display immunosuppressive, nephrotoxic, or carcinogenic effects, among others.[11] According to their carcinogenic potential, some mycotoxins, like aflatoxins, have been included in the classification list of human carcinogens provided by the International Agency for Research on Cancer (IARC).[12] These mycotoxins are produced by the genera Aspergillus, which has been categorized as a major fungus occurring in C. sativa inflorescences alongside other mycotoxin producing fungi, like Fusarium spp., so different mycotoxins could be also expected.[13][14] On the other hand, pesticides include a broad range of compounds routinely applied to protect crops from different pests. However, residues coming from these products can accumulate in plants intended for human consumption, leading to several health issues related to neurotoxicity, carcinogenicity, and pulmonotoxicity, as well as developmental and reproductive disorders.[15][16][17][18]\nIn terms of regulation, maximum residue limits (MRL) for different types of contaminants have been set by the E.U. Regulation (EC) No. 396\/2005[19] establishes limits for pesticides, whereas Regulation (EC) No. 1881\/2006[20] covers mycotoxins, attaching maximum limits in food and feeds. Nevertheless, nutraceutical products are not considered by the legislation yet; however, due to a potential carryover during the manufacturing process, contamination could be expected not only in raw material, but also in other by-products. Moreover, several studies have reported the sole presence of pesticides[21][22], mycotoxins[23][24], and both types of contaminants[25][26][27] in diverse food supplements, emphasizing the necessity to evaluate the contamination profile of these products considering their rising consumption and popularity.\nTo overcome this point, the development of analytical procedures is needed. Concerning the extraction of contaminants, QuEChERS (quick, easy, cheap, effective, rugged and safe)[21][23][24] and \u201cdilute and shoot\u201d procedures have been recently applied to food supplements delivered as gelatin capsules, traditional capsules, tablets, powder extracts or liquid presentations.[25][26][27] Analytical methods used in the detection and quantification of contamination include ELISA detection[28], gas chromatography (GC) coupled with mass spectrometry (MS)[22] and ultra-high-performance liquid chromatography (UHPLC) coupled with tandem mass spectrometry (MS\/MS)[23][24], and high-resolution Orbitrap mass spectrometry (Q-Orbitrap HRMS).[25][26][27] Due to its high resolving power, sensitivity and accurate mass measurement, high-resolution mass spectrometry stands as a suitable alternative for evaluating a large number of contaminants present in complex matrices at low concentrations. Therefore, the aim of the present study was to provide an analysis of pesticide residues and mycotoxins produced by major C. sativa fungi occurring in CBD-based food supplements using ultra-high-performance liquid chromatography coupled with high-resolution Orbitrap mass spectrometry. To achieve this, a novel methodology was developed in order to identify and quantify 16 mycotoxins after evaluating different extraction procedures, followed by a post-target screening of 283 pesticides based on a comprehensive spectral library. To the best of the authors\u2019 knowledge, this is the first multi-class analysis of CBD-based supplements through the use of high-resolution mass spectrometry techniques.\n\nResults \nOptimization of extraction procedure \nThe molecular complexity of this matrix demands an effective extraction in order to detect and quantify several mycotoxins in a reliable way. A QuEChERS methodology previously developed on this typology of sample[24] was selected as the starting point, whereas different volumes of extraction solvent and the type of sorbent for clean-up was tested.\n\nEvaluation of the volume of extraction solvent \nThe extraction procedure was first evaluated in triplicate by spiking the sample at 10 ng\/g using the following volumes of extraction solvent per gram of sample: 2.5, 5, 7.5 and 10 mL.\nThe extraction performed with 2.5 mL showed recovery values below the minimum limit (70%) for the vast majority of the studied analytes as a consequence of solvent saturation (Figure 1a). Satisfactory recoveries (70\u2013120%) were obtained after performing the extraction with 5 mL of solvent for the majority of compounds, with the exception of \u03b2-ZEL (155%) and ZAN (150%), which were significantly more efficient than the other volumes tested (p < 0.05). On the other hand, the extractions performed with 7.5 and 10 mL showed a gradual decrease in recoveries due to the larger dilution of the analytes. Therefore, 5 mL of AcN was selected as the optimal volume of extraction solvent for this type of CBD capsule.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 Percentage of mycotoxins extracted with a recovery value (R) below 70% (white), between 70% and 120% (black) and above 120% (grey), corresponding to extractions performed with: (a) different volumes of solvent at a spiking level of 10 ng\/g; (b) different sorbents for clean-up at a spiking level of 10 ng\/g.\n\n\n\nEvaluation of the type of sorbent for clean-up \nThe molecular composition of the soft gel capsules mainly consists on fatty acids and proteins. Because of the complex nature of this matrix, an efficient clean-up is required in order to avoid interference with the analytes. To achieve this, clean-up with different sorbents (100 mg) was performed, including octadecyl carbon chain-bonded silica (C18) (as previously suggested[24]), graphitized carbon black (GCB), primary-secondary amine (PSA), and zirconium oxide (Z-Sep+).\nPSA exhibited a good performance for the vast majority of analytes (Figure 1b, above) but was unable to recover other important mycotoxins, such as AFB1 and AFG1. The moderate affinity of PSA with polar compounds may explain low recoveries for aflatoxins, being consistent with other works based on oily matrices.[29][30] Similarly, extraction with C18 was efficient for most compounds, and only some low-polarity mycotoxins showed recoveries out of the range set, like ZAN (150%) and \u03b2-ZEL (155%). Clean-up using GCB showed poor results, allowing us to detect only NEO (85%), HT-2 (89%), and T-2 (89%). This sorbent is able to retain planar molecules; additionally, mycotoxin adsorption has been previously reported[31], which might be the reason for the low recoveries obtained here. Finally, extraction performed with Z-Sep+ showed satisfactory recoveries (70\u2013120%) for all the mycotoxins studied.\nOn the other hand, the influence of the matrix was minimal (80% \u2264 signal suppression\/enhancement (SSE) \u2264 120%) for all targeted analytes when using Z-Sep+ and PSA. Clean-up based on Z-Sep+ has been successfully applied to the extraction of analytes from lipid matrices.[32][33] Furthermore, Z-Sep+ is also able to form irreversible links with carboxylic groups present in proteins[34], standing as the most suitable sorbent for the here-analyzed matrix. Similarly, the use of PSA has been suggested to remove coextracted fatty acids and other ionic lipids.[35]\nOn the contrary, a strong matrix effect was evidenced for half the analytes when using C18 and GCB. Signal suppression was detected after using C18, obtaining SSE ranging from 40% to 69%, whereas signal enhancement occurred after GCB clean-up, with SSE increasing from 128% to 167%. Since both sorbents have a preferential affinity for non-polar compounds, matrix interferents were not fully removed but coextracted. The presence of these coextracted species can change the ionization efficiency, leading to improper SSE and an unreliable quantification. Although no significant differences were observed between the use of Z-Sep+ and PSA (p > 0.05), Z-Sep+ was chosen because of its better performance minimizing matrix interference.\n\nAnalytical method validation \nThe optimized method was validated for the simultaneous extraction of 16 mycotoxins in CBD-based products. Results are shown in Table 1. Good linearity was observed for all analytes in the range assessed (0.20\u2013100 ng\/g), with regression coefficients (r2) above 0.990 and a deviation \u226420% for each level of the calibration curve. Comparison between calibration curves built in a blank matrix and in neat solvent showed a minimal interference in the matrix (\u00b120%) for the studied analytes. Hence, external calibration curves were used for quantification purposes. Limits of quality (LOQs) obtained for all studied analytes were between 0.20 and 6.25 ng\/g. Regarding trueness, recovery values corresponding to a fortification level of 20 ng\/g ranged between 63 and 103% and between 63 and 113% for the lowest fortification level (10 ng\/g). Referring to the additional spiking level (2 ng\/g) for aflatoxins, recoveries ranged between 63% and 86%. Precision study revealed both RSDr and RSDR values below 20% for all the mycotoxins analyzed. These results confirmed that the optimized procedure is suitable for a reliable quantification of the mycotoxins analyzed, fulfilling the criteria set by Commission Decision 2002\/657\/EC.[36] \n\r\n\n\n\n\n\n\n\n\n\n\n Table 1 Method performance: linearity, matrix effect (SSE %), recovery and LOQ. (1 Additional fortification level only for AFs)\n\n\n\nTable 2 reviews the available literature regarding mycotoxins in herbal-based supplements. As shown, the here-obtained LOQs were lower than the ones reported in previous studies using UHPLC-Q-Orbitrap HRMS. As established by Regulation (EC) No. 1881\/2006[20], maximum limits for aflatoxins in many food matrices must not reach levels which are below those LOQs (5 ng\/g), whereas LOQs obtained in this study were between 5 and 25 times lower. Other analytical methods based on low resolution mass spectrometry[37] required longer and more complicated extraction procedures than the QuEChERS developed here. Even ELISA detection has been used for quantification of mycotoxins in medicinal herbs[28], but a very specific extraction had to be performed for different groups of analytes using several multi-functional columns. The QuEChERS procedure developed in this study, in combination with UHPLC-Q-Orbitrap mass spectrometry, was extremely simple and reliable, allowing for the quantification of all mycotoxins with high sensitivity.\n\n\n\n\n\n\n\nTable 2. Available methods for measurement of mycotoxins in herbal-based supplements 1\r\n \r\n1 ESI+ = positive ion mode; ESI\u2212 = negative ion mode; HRMS = high-resolution MS; LOQ = limit of quantification; MRM = multiple reaction monitoring; QQQ = triple quadrupole. 2 Range of LOQs referring to the analyzed mycotoxins. 3 AFs = aflatoxins; DON = deoxynivalenol; CIT = citrinin; FB1 = fumonisin B1; MPA = mycophenolic acid; OTA = ochratoxin A; OTB = ochratoxin B. NEO = neosolaniol; ZEN = zearalenone; ENNA, ENNA1, ENNB, and ENNB1 = enniatins.\n\n\nSamples procedence (no.)\n\nPositives samples (%)\n\nMajor analytes detected\n\nConcentration reported (ng\/g)\n\nDetermination\n\n\nSensitivity (LOQ, ng\/g)\n\nDetection method\n\nReference\n\n\nMedicinal or aromatic herbs (84)\n\n99\n\nZEN\n\n1.0\u201344.1\n\n0.14\n\nELISA detection (EIA reader, SIRIO S)\n\n[28]\n\n\nT-2\n\n0.6\u2013256.9\n\n0.28\n\n\nDON3\n\n20.5\u2013343.5\n\n14.8\n\n\nCIT3\n\n14.9\u2013354.8\n\n16.5\n\n\nTraditional Chinese herbs (60)\n\n83\n\nZEN\n\n2.1\u201315.5\n\n0.4\n\nQQQ (Applied Biosystems) ESI+ MRM mode\n\n[37]\n\n\nAFs3\n\n0.2\u201319.5\n\n0.1\n\n\nMPA3\n\n0.2\u201322.7\n\n0.02\n\n\nMilk thistle (83)\n\n19\n\nAFB1\n\n0.04\u20131.9\n\n0.03\n\nLC-FLD (Waters)\n\n[38]\n\n\n\nGreen coffee bean (50)\n\n36\n\nOTA3\n\n1\u2013136.9\n\n2.5\n\nQQQ (AB SCIEX) ESI+ and ESI- MRM mode\n\n[23]\n\n\nOTB3\n\n1\u201320.2\n\n2.5\n\n\nFB13\n\n50\u2013415\n\n100\n\n\nMPA\n\n5\u2013395\n\n10\n\n\nMilk thistle (7)\n\n29\n\nT-2\n\n363\u2013453.9\n\n30.5\n\nQQQ (AB SCIEX) ESI+ MRM mode\n\n[39]\n\n\nHT-2\n\n826.9\u2013943.7\n\n43.8\n\n\nHerbals (69)\n\n96\n\nZEN\n\n5\u2013824\n\n10\n\nQQQ (AB SCIEX) ESI+ and ESI- MRM mode\n\n[24]\n\n\nT-2\n\n69\u20131870\n\n10\n\n\nHT-2\n\n59\u20131530\n\n50\n\n\nENNB\n\n5\u20139260\n\n5\n\n\nENNB1\n\n5\u201310,900\n\n5\n\n\nENNA\n\n5\u20138340\n\n5\n\n\nENNA1\n\n5\u20132340\n\n5\n\n\nGinkgo biloba (8)\n\n50\n\nAFB1\n\n5.0\u201354\n\n5\n\nQ-Orbitrap (Exactive, Thermo FisherScientific) ESI+ and ESI- HRMS\n\n[25]\n\n\nAFB2\n\n4\u2013300\n\n10\n\n\nT-2\n\n18\u201320\n\n30.5\n\n\nGreen tea (10)\n\n10\n\nAFB1\n\n5.4\n\n5\n\nQ-Orbitrap (Exactive, Thermo FisherScientific) ESI+ and ESI- HRMS\n\n[26]\n\n\nRoyal jelly (8)\n\n0\n\n\nSoy (11)\n\n27\n\nAFB1\n\n8.2\u201317.1\n\n5\n\nQ-Orbitrap (Exactive, Thermo FisherScientific) ESI+ and ESI- HRMS\n\n[27]\n\n\nAFG2\n\n6.4\n\n5\n\n\nCannabis sativa (10)\n\n70\n\nZEN\n\n4.2\u201311.6\n\n3.13\n\nQ-Orbitrap (Exactive, Thermo FisherScientific) ESI+ and ESI- HRMS\n\nCurrent study\n\n\nENNB1\n\n<LOQ\u201311.6\n\n1.56\n\n\n\nApplication to commercial CBD-based products \nThe validated UHPLC-Q-Orbitrap HRMS procedure was applied to 10 commercially available samples in order to evaluate the occurrence of mycotoxins. Results are shown in Table 3. A considerable occurrence of mycotoxins was observed, since contamination with at least one analyte was found in 70% of the samples. Up to six different mycotoxins (T-2, ZAN, ZEN, ENNB1, ENNA, ENNA1) were quantified at a range from below LOQ to 11.6 ng\/g, all produced by Fusarium genera, reported as a major C. sativa pathogen fungus.[14] Previous studies regarding mycotoxins in different herbal-based extracts have revealed the occurrence of similar mycotoxins independently of the matrix and the dosage form (Table 2, above). Despite the fact that the percentage of positive samples varied among the different studies (19\u201399%), when the sensitivity of the analytical method increased, reaching lower LOQs, the number of positive samples dramatically increased. This indicated that mycotoxin contamination in herbal-based products at low levels is frequent.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 3 Occurrence of studied mycotoxins in the analyzed samples\n\n\n\nIn the here-analyzed samples, ZEN appeared to be the most common mycotoxin, with an incidence of 60% and concentration levels ranging from 4.2 to 11.6 ng\/g (mean level = 6.9 ng\/g). A high incidence of ZEN has also been previously reported in supplements made of different herbals from Czech and U.S. retail markets (84%, n = 69) at a wide range of concentrations (5\u2013824 ng\/g, mean value = 75.7 ng\/g).[24] Moreover, ZEN was previously found in 96% of medicinal herbals from Spain (n = 84) as well, but in a tighter range (1\u201344.1 ng\/g, mean value = 8.9 ng\/g).[28]\nReferring to T-2, results reported contamination in one sample at 2.0 ng\/g, in contrast with the prevalent presence of T-2 in 78% (n = 69) of the same Czech and U.S. samples, at concentrations rising from 69 to 1,870 ng\/g (mean value = 162 ng\/g).[24] High levels of T-2 were also observed in milk thistle samples from Spain (363\u2013453.9, mean value = 408.9 ng\/g) in only two out of seven samples.[39] In the other hand, T-2 was quantified in 98% (n = 84) of the Spanish medicinal herbals, but in much lower concentrations (0.6\u2013256 ng\/g, mean value = 22.645 ng\/g).[28]\nSimilarly, ZAN was quantified in one sample at 1.9 ng\/g. This mycotoxin has been scarcely targeted in dietary supplement studies, but it has been previously quantified at similar concentrations as those here-reported in two samples of Chinese medicinal herbals (n = 33).[40]\nResults also showed ENN contamination. ENNB1, ENNA, and ENNA1 were found in the same sample at 11.6, 4.2 and 5.8 ng\/g, respectively, whereas ENNB1 was detected in two other samples below the LOQ (1.56 ng\/g). These emerging Fusarium mycotoxins have been previously found in herbal products (84\u201391%, n = 69) widely ranging from 5 ng\/g up to 10,900 ng\/g (mean value = 354 ng\/g).[24] Similarly, ENNB1 was the most common toxin out of these emerging Fusarium mycotoxins, being consistent with the results here obtained.\nAll the mycotoxins found in the present study correspond to low- to non-polar compounds, which should be prevalently expected due to the nature of the matrix.\nCo-occurrence of at least two mycotoxins was also observed in four out of 10 samples. Results showed the presence of ZEN in combination with ENNs B1, A and A1, ZAN, or T-2, which are common associations found by previous studies in herbal-based supplements.[24][28] It must be noted that synergic or additive effects have been observed as a consequence of these combinations in in vitro assays.[41] Based on what has been discussed and considering the uprising trend of C. sativa-based products, alongside the use of environment-friendly raw materials cultivated without pesticides, quality controls regarding mycotoxins should be set for these products in order to ensure safe consumption.\n\nIdentification of non-target compounds through retrospective analysis in studied samples \nThe post-target screening approach allowed us to detect pesticide residues in the analyzed samples using a spectral library. Results are shown in Figure 2. Up to 46 different pesticides were tentatively identified based on the mass spectral library of the pesticides. Ethoxyquin was putatively found in five samples, being the most prevalent pesticide. The main function of ethoxyquin is to avoid fungal contamination during the postharvest stage of the plant through its scald-preventive properties.[42] Surprisingly, the use of this pesticide is forbidden by the European Commission's (EC) Decision 2011\/143\/EU. Piperonyl butoxide was found in four samples. This compound is not a pesticide by itself but can inhibit the resistance mechanisms of insects, being widely used in combination with other different pesticides.[42] The tentative presence of simazine and cyanazine, both found in three different samples, must also be noted. The use of these pesticides was prohibited by EC Regulation No. 1107\/2009[43] and Commission Decision 2004\/247\/EC[44], respectively. Therefore, the occurrence of forbidden pesticides found in the here-analyzed samples highlights the necessity of monitoring potential contaminants in C. sativa-derived products acquired from online shops.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2 Occurrence of non-target pesticides in analyzed samples after post-run retrospective screening.\n\n\n\nConclusions \nA sample preparation procedure based on a QuEChERS followed by UHPLC coupled with high-resolution Q-Orbitrap mass spectrometry was optimized in order to determine and quantify 16 mycotoxins in C. sativa-based supplements. The proposed methodology was validated following the E.U. criteria, ensuring a proper specificity, selectivity, linearity, trueness, and precision with a fast chromatography run performance (eight minutes). The validated procedure was applied to ten CBD-based supplements that are commercially available online, allowing us to quantify up to six different Fusarium mycotoxins in 70% of samples. ZEN was the most prevalent mycotoxin (60%) found at a maximum level of 11.6 ng\/g (mean value = 6.9 ng\/g). Co-occurrence was observed in four out of ten samples, including one sample with ENNB1, ENNA, and ENNA1. Additionally, a retrospective analysis of pesticide residues was performed. Up to 46 different pesticides were tentatively detected, including some forbidden in C. sativa cultivation. Considering the uprising trend of CBD-based products, quality controls regarding contaminants should be set for these products in order to ensure a safe consumption. Furthermore, the developed procedure is proposed as a powerful analytical tool to evaluate the potential mycotoxin profile of these particular products.\n\nMaterials and methods \nChemicals and reagents \nAcetonitrile (AcN), methanol (MeOH), and water for LC mobile phase (HPLC grade) were acquired from Merck (Darmstadt, Germany). Formic acid and ammonium formate were obtained from Fluka (Milan, Italy). Sodium chloride (NaCl), magnesium sulfate (MgSO4), octadecyl carbon chain-bonded silica (C18), graphitized carbon black (GCB), primary-secondary amine (PSA), and zirconium oxide (Z-Sep+) were obtained from Sigma Aldrich (Milan, Italy).\nMycotoxin standards and metabolites, namely aflatoxins (AFB1, AFB2, AFG1, and AFG2), HT-2 toxin (HT-2), T-2 toxin (T-2), neosolaniol (NEO), zearalenone (ZEN), \u03b1-zearalenol (\u03b1-ZEL), \u03b2-zearalenol (\u03b2-ZEL), zearalanone (ZAN), beauvericin (BEA) and enniatins (ENNA, ENNA1, ENNB, and ENNB1) were purchased from Sigma Aldrich (Milan, Italy). Individual stock solutions of all analytes were prepared by diluting 1 mg of each mycotoxin in 1 mL of methanol. The working standard solution including all the mycotoxins was made by adequate diluting in MeOH:H2O (70:30 v\/v) 0.1% formic acid to reach the required concentrations for performing the spike experiments: 20, 10 and 2 \u00b5g\/mL. All solutions were kept in safe conditions at \u221220 \u00b0C.\n\nSampling \nFor the analysis of real samples, 10 different CBD gelatin capsules were obtained from online shops based in different European countries. The capsules are made of gel mass, which contains gelatin, water, glycerin, and other minor additives, whereas the fill formulation consists of olive oil mixed with hemp oil containing CBD at certain concentrations. The weight of each capsule depended on the manufacturer; there were 0.25, 0.5, and 1 g capsules. Only soft gel capsules were studied since it was the prevalent presentation available for CBD supplements. On the other hand, one sample of CBD supplements delivered as soft gel capsules was acquired from a local store (Naples, Italy). After confirming the absence of contaminants, they were used for preparing fortified samples for recovery assays and matrix-matched standards for calibration purposes. All the samples were conserved in dark and cool conditions, as recommended by the manufacturer, until further analysis.\n\nSample preparation \nThe sample preparation procedure developed by Veprikova et al.[24] was selected as a starting point and then slightly modified, as follows: 1 g of sample was weighed into a 50 mL polytetrafluorethylene (PTFE) tube and mixed with 5 mL of 1% aqueous formic acid. The mixture was placed in an SKO-D XL orbital shaker (Argo Lab, Italy) for 30 minutes at 294 \u00d7 g. Then, 5 mL of AcN were added and the mixture was shaken for an additional 30 minutes at 294 \u00d7 g. After that, 0.5 g of sodium chloride and 2 g of magnesium sulfate were added and the tube was shaken for 1 minute by hand, followed by centrifugation at 4907 \u00d7 g for 15 minutes in an SL 16R centrifuge (Thermo Fisher Scientific LED GmbH, Germany). A 2 mL aliquot of the upper acetonitrile layer was taken for dispersive solid phase extraction (d-SPE) cleanup in a 15 mL PTFE tube containing 100 mg of Z-Sep+ sorbent and 300 mg of magnesium sulfate. The tube was vortexed for 1 minute and then centrifuged at 4907 \u00d7 g for 15 min. An aliquot of the supernatant (1 mL) was collected and filtered through a 0.2 \u00b5m PTFE filter (Phenomenex, Italy) into a vial prior to UHPLC-Q-Orbitrap HRMS analysis.\n\nUHPLC-Q-Orbitrap HRMS analysis \nThe qualitative and quantitative profiles of the mycotoxins were obtained using an ultra-high-pressure liquid chromatograph (UHPLC, Thermo Fisher Scientific, Waltham, MA, USA) equipped with a degassing system, a Dionex Ultimate 3000, a Quaternary UHPLC pump working at 1250 bar, an auto sampler device, and a thermostated (T = 30 \u00b0C) Luna Omega 1.6 \u00b5m (50 \u00d7 2.1 \u00b5m) column.\nThe eluent consisted of two different phases: A.) (H2O containing 0.1% formic acid and 5 mM ammonium formate) and B.) (MeOH containing 0.1% formic acid and 5 mM ammonium formate). The gradient elution for LC-Orbitrap HRMS analyses was applied as follows: an initial 0% of phase B was held for 1 minute, which linearly went up to 95% B over 1 minute and held for 0.5 minutes. Next, the gradient decreased to 75% B over 2.5 minutes and then decreased again to 60% B over 1 minute. Finally, the gradient turned to 0% B over 0.5 minutes and then the column was equilibrated for 1.5 minutes at 0% B. The total run time was 8 minutes, at a flow rate of 0.4 mL\/min. A total of 5 \u00b5L of the sample was injected. Detection was performed using a Q-Exactive mass spectrometer. The mass spectrometer was operated in both positive and negative ion mode using fast polarity switching by setting two scan events (full ion MS and all ion fragmentation (AIF)). Full scan data were acquired at a resolving power of 35,000 FWHM at m\/z 200.\nThe ion source parameters were: spray voltage 4 kV (-4 kV in ESI\u2212 mode); capillary temperature 290 \u00b0C; S-lens RF level 50; sheath gas pressure (N2 > 95%) 35, auxiliary gas (N2 > 95%) 10, and auxiliary gas heater temperature 305 \u00b0C. The value for automatic gain control (AGC) target was set at 1 \u00d7 106, a scan range of m\/z 100 to 1000 was selected and the injection time was set to 200 ms. The scan rate was set at 2 scans\/s. For the scan event of AIF, the parameters in the positive and negative ion mode were: mass resolving power = 17,500 FWHM; maximum injection time = 200 ms; scan time = 0.10 s; ACG target = 1 \u00d7 105; scan range = 100\u20131000 m\/z, isolation window to 5.0 m\/z, and retention time window to 30 seconds. The Orbitrap-MS parameters were optimized in a previous work.[45] The exact mass for the studied compounds, including elemental composition, retention time (RT), theoretical masses and accurate mass errors for the detected ions are shown in Table 4. A mass error below 5 ppm, referring to the molecular ions, was set for identification. Retrospective screening was carried out on spectral data collected using a pesticide spectral library (Pesticide Spectral Library Version 1.1 for LibraryView\u2122 Software, AB SCIEX, Framingham, USA). For accurate mass measurement, identification and confirmation were performed at a mass tolerance of 5 ppm for the molecular ion and for both fragments at the intensity threshold of 1000. Data analysis and processing were performed using the Xcalibur software, v. 3.1.66.10.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 4. Retention times, accurate mass and mass accuracy of mycotoxins evaluated\n\n\n\nValidation parameters \nAn in-house validation study was conducted following the E.U. Commission Decision 2002\/657\/EC.[36] The parameters evaluated were selectivity, specificity, linearity, trueness, repeatability (intra-day precision), within-reproducibility (inter-day precision), limit of quantification (LOQ), and limit of detection (LOD). The selectivity and specificity of the method were evaluated by analyzing both standard solutions and samples, comparing the retention time of the peaks corresponding to the analytes of interest alongside the determination of its precursor and product ion, with a mass error below 5 ppm. For linearity, standard solutions built in neat solvent and matrix-matched calibration were analyzed by spiking blank samples at eight concentration levels from 0.2 to 100 ng\/g. The slopes of each linear calibration function were compared in order to detect a signal suppression\/enhancement (SSE) effect due to the matrix interference. This effect was quantified following the equation: SSE (%) = matrix-matched calibration slope\/solvent calibration slope x 100. An SSE value of 100% was interpreted as no matrix interference in the concentration range evaluated. An SSE value above 100% revealed signal enhancement whereas a value below 100% indicated signal suppression. For trueness, recovery studies were evaluated by spiking three blank samples at three different levels. Additionally, a lower spike level was used only for aflatoxins. Intra-day precision (RSDr) was expressed as the relative standard deviation after three determinations in a single day (n = 3). Inter-day precision was calculated by repeating the measurements in triplicate on three non-consecutive days (n = 9) and expressed as relative standard deviation (RSDR). The LOD was defined as the minimum concentration where the molecular ion can be identified by the instrument (mass error value below 5 ppm) and the LOQ as the minimum concentration where a linear response (mass error value below 5 ppm) can be observed with an accuracy and precision of \u2264 20%.\n\nStatistical analysis \nValidation experiments were performed in triplicate and the results expressed as the average values alongside relative standard deviation (RSD, %). The Saphiro\u2013Wilk test was applied to evaluate normality and multivariant analysis was performed using a non-parametric Kruskal\u2013Wallis test, considering p values < 0.05 as significant. Analysis of data was carried out using IBM SPSS version 25 statistical software package (SPSS, Chicago, IL, USA).\n\nAcknowledgements \nThe authors want to acknowledge Alice Valli for donating sample material.\n\nAuthor contributions \nConceptualization, Y.R.-C. and A.N.; methodology, L.I. and A.N.; validation, L.C. and A.N.; formal analysis, A.N.; investigation, L.C., L.I. and A.N.; resources, A.R.; writing\u2014original draft preparation, A.N.; writing\u2014review and editing, Y.R.-C.; supervision, Y.R.-C. and A.R.; project administration, Y.R.-C. and A.R.; funding acquisition, A.R. All authors have read and agreed to the published version of the manuscript.\n\nFunding \nThis research received no external funding.\n\nConflicts of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 Afshin, A.; Sur, P.J.; Fay, K.A. et al. (2019). \"Health effects of dietary risks in 195 countries, 1990-2017: A systematic analysis for the Global Burden of Disease Study 2017\". Lancet 393 (10184): 1958\u201372. doi:10.1016\/S0140-6736(19)30041-8. PMC PMC6899507. PMID 30954305. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6899507 .   \n\n\u2191 Almada, A.L. (2019). \"Chapter 1: Nutraceuticals and functional foods: Innovation, insulation, evangelism, and evidence\". In Bagchi, D.. Nutraceutical and Functional Food Regulations in the United States and around the World (3rd ed.). Academic Press. pp. 3\u201311. ISBN 9780128164679.   \n\n\u2191 Binns, C.W.; Lee, M.K.; Lee, A.H. (2018). \"Problems and Prospects: Public Health Regulation of Dietary Supplements\". Annual Review of Public Health 39: 403\u201320. doi:10.1146\/annurev-publhealth-040617-013638. PMID 29272167.   \n\n\u2191 Zuardi, A.W. (2008). \"Cannabidiol: From an inactive cannabinoid to a drug with wide spectrum of action\". Brazilian Journal of Psychiatry 30 (3): 271\u201380. doi:10.1590\/s1516-44462008000300015. PMID 18833429.   \n\n\u2191 Casares, L.; Garc\u00eda, V.; Garrido-Rodr\u00edguez, M. et al. (2020). \"Cannabidiol induces antioxidant pathways in keratinocytes by targeting BACH1\". Redox Biology 28: 101321. doi:10.1016\/j.redox.2019.101321. PMC PMC6742916. PMID 31518892. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6742916 .   \n\n\u2191 Maroon, J.; Bost, J. (2018). \"Review of the neurological benefits of phytocannabinoids\". Surgical Neurology International 9: 91. doi:10.4103\/sni.sni_45_18. PMC PMC5938896. PMID 29770251. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5938896 .   \n\n\u2191 \"Regulation (EU) 2015\/2283 of the European Parliament and of the Council of 25 November 2015 on novel foods, amending Regulation (EU) No 1169\/2011 of the European Parliament and of the Council and repealing Regulation (EC) No 258\/97 of the European Parliament and of the Council and Commission Regulation (EC) No 1852\/2001\". EUR-Lex. European Union. 11 December 2015. https:\/\/eur-lex.europa.eu\/legal-content\/en\/TXT\/?uri=CELEX%3A32015R2283 .   \n\n\u2191 \"Iternational CBD and Cannabis Market Landscape\". Brightfield Group Data Portals. Brightfield Group. https:\/\/www.brightfieldgroup.com\/international-cbd-cannabis-market-landscape .   \n\n\u2191 Santini, A.; Cammarata, S.M.; Capone, G. et al. (2018). \"Nutraceuticals: Opening the debate for a regulatory framework\". British Journal of Clinical Pharmacology 84 (4): 659\u2013672. doi:10.1111\/bcp.13496. PMC PMC5867125. PMID 29433155. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5867125 .   \n\n\u2191 Gulati, O.P.; Ottaway, P.B.; Jenning, S. et al. (2019). \"Chapter 20: Botanical nutraceuticals (food supplements and fortified and functional foods) and novel foods in the EU, with a main focus on legislative controls on safety aspects\". In Bagchi, D.. Nutraceutical and Functional Food Regulations in the United States and around the World (3rd ed.). Academic Press. pp. 277\u2013321. ISBN 9780128164679.   \n\n\u2191 Rodr\u00edguez-Carrasco, Y.; Fattore, M.; Albrizio, S. et al. (2015). \"Occurrence of Fusarium mycotoxins and their dietary intake through beer consumption by the European population\". Food Chemistry 178: 149\u201355. doi:10.1016\/j.foodchem.2015.01.092. PMID 25704695.   \n\n\u2191 Ostry, V.; Malir, F.; Toman, J. et al. (2017). \"Mycotoxins as human carcinogens-the IARC Monographs classification\". Mycotoxin Research 33 (1): 65\u201373. doi:10.1007\/s12550-016-0265-7. PMID 27888487.   \n\n\u2191 McKernan, K.; Spangler, J.; Zhang, L. et al. (2015). \"Cannabis microbiome sequencing reveals several mycotoxic fungi native to dispensary grade Cannabis flowers\". F1000Research 4: 1422. doi:10.12688\/f1000research.7507.2. PMC PMC4897766. PMID 27303623. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4897766 .   \n\n\u2191 14.0 14.1 McHardy, I.; Romanelli, A.; Harris, L.J. et al. (2018). \"Infectious risks associated with medicinal Cannabis: Potential implications for immunocompromised patients?\". Journal of Infection 76 (5): 500\u20131. doi:10.1016\/j.jinf.2018.01.010. PMID 29408325.   \n\n\u2191 Atapattu, S.N.; Johnson, K.R.D. (2020). \"Pesticide analysis in cannabis products\". Journal of Chromatography A. 1612: 460656. doi:10.1016\/j.chroma.2019.460656. PMID 31679712.   \n\n\u2191 Choudri, B.S.; Charabi, Y. (2019). \"Pesticides and herbicides\". Water Environment Research 91 (10): 1342-1349. doi:10.1002\/wer.1227. PMID 31523896.   \n\n\u2191 Mostafalou, S.; Abdollahi, M. (2017). \"Pesticides: An update of human exposure and toxicity\". Archives of Toxicology 91 (2): 549-599. doi:10.1007\/s00204-016-1849-x. PMID 27722929.   \n\n\u2191 Ye, M.; Beach, J.; Martin, J.W. et al. (2017). \"Pesticide exposures and respiratory health in general populations\". Journal of Environmental Sciences 51: 361\u201370. doi:10.1016\/j.jes.2016.11.012. PMID 28115149.   \n\n\u2191 \"Regulation (EC) No 396\/2005 of the European Parliament and of the Council of 23 February 2005 on maximum residue levels of pesticides in or on food and feed of plant and animal origin and amending Council Directive 91\/414\/EECText with EEA relevance\". EUR-Lex. European Union. 16 March 2005. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32005R0396 .   \n\n\u2191 20.0 20.1 \"Commission Regulation (EC) No 1881\/2006 of 19 December 2006 setting maximum levels for certain contaminants in foodstuffs\". EUR-Lex. European Union. 20 December 2006. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32006R1881 .   \n\n\u2191 21.0 21.1 Jeong, M.L.; Zahn, M.; Trinh, T. et al. (2008). \"Pesticide residue analysis of a dietary ingredient by gas chromatography\/selected-ion monitoring mass spectrometry using neutral alumina solid-phase extraction cleanup\". Journal of AOAC International 91 (3): 630-6. PMID 18567310.   \n\n\u2191 22.0 22.1 Gonz\u00e1lez-Mart\u00edn, M.I.; Revilla, I.; Betances-Salcedo, E.V. et al. (2018). \"Pesticide residues and heavy metals in commercially processed propolis\". Microchemical Journal 143: 423\u201329. doi:10.1016\/j.microc.2018.08.040.   \n\n\u2191 23.0 23.1 23.2 23.3 Vaclavik, L.; Vaclavikova, M.; Begley, T.H. et al. (2013). \"Determination of multiple mycotoxins in dietary supplements containing green coffee bean extracts using ultrahigh-performance liquid chromatography-tandem mass spectrometry (UHPLC-MS\/MS)\". Journal of Agricultural and Food Chemistry 61 (20): 4822-30. doi:10.1021\/jf401139u. PMID 23631685.   \n\n\u2191 24.00 24.01 24.02 24.03 24.04 24.05 24.06 24.07 24.08 24.09 24.10 Veprikova, Z.; Zachariasova, M;. Dzuman, Z. et al. (2015). \"Mycotoxins in Plant-Based Dietary Supplements: Hidden Health Risk for Consumers\". Journal of Agricultural and Food Chemistry 63 (29): 6633-43. doi:10.1021\/acs.jafc.5b02105. PMID 26168136.   \n\n\u2191 25.0 25.1 25.2 25.3 Mart\u00ednez-Dom\u00ednguez, G.; Romero-Gonz\u00e1lez, R.; Garrido Frenich, A. (2015). \"Determination of toxic substances, pesticides and mycotoxins, in ginkgo biloba nutraceutical products by liquid chromatography Orbitrap-mass spectrometry\". Microchemical Journal 118: 124\u201330. doi:10.1016\/j.microc.2014.09.002.   \n\n\u2191 26.0 26.1 26.2 26.3 Mart\u00ednez-Dom\u00ednguez, G.; Romero-Gonz\u00e1lez, R.; Garrido Frenich, A. (2016). \"Multi-class methodology to determine pesticides and mycotoxins in green tea and royal jelly supplements by liquid chromatography coupled to Orbitrap high resolution mass spectrometry\". Food Chemistry 197 (Pt. A): 907-15. doi:10.1016\/j.foodchem.2015.11.070. PMID 26617033.   \n\n\u2191 27.0 27.1 27.2 27.3 Mart\u00ednez-Dom\u00ednguez, G.; Romero-Gonz\u00e1lez, R.; Arrebola, F.J. et al. (2016). \"Multi-class determination of pesticides and mycotoxins in isoflavones supplements obtained from soy by liquid chromatography coupled to Orbitrap high resolution mass spectrometry\". Food Control 59: 218\u201324. doi:10.1016\/j.foodcont.2015.05.033.   \n\n\u2191 28.0 28.1 28.2 28.3 28.4 28.5 Santos, L.; Mar\u00edn, S.; Sanchis, V. et al. (2009). \"Screening of mycotoxin multicontamination in medicinal and aromatic herbs sampled in Spain\". Journal of the Science of Food and Agriculture 89 (10): 1802\u20137. doi:10.1002\/jsfa.3647.   \n\n\u2191 Zhao, H.; Chen, X.; Shen, C. et al. (2017). \"Determination of 16 mycotoxins in vegetable oils using a QuEChERS method combined with high-performance liquid chromatography-tandem mass spectrometry\". Food Additives and Contaminates Part A 34 (2): 255\u201364. doi:10.1080\/19440049.2016.1266096. PMID 27892850.   \n\n\u2191 Hidalgo-Ruiz, J.L;. Romero-Gonz\u00e1lez, R.; Mart\u00ednez Vidal, J.L. et al. (2019). \"A rapid method for the determination of mycotoxins in edible vegetable oils by ultra-high performance liquid chromatography-tandem mass spectrometry\". Food Chemistry 288: 22\u20138. doi:10.1016\/j.foodchem.2019.03.003. PMID 30902285.   \n\n\u2191 Myresiotis, C.K.; Testempasis, S;. Vryzas, Z. et al. (2015). \"Determination of mycotoxins in pomegranate fruits and juices using a QuEChERS-based method\". Food Chemistry 182: 81\u20138. doi:10.1016\/j.foodchem.2015.02.141. PMID 25842312.   \n\n\u2191 Han, L;. Matarrita, J.; Sapozhnikova, Y. et al. (2015). \"Evaluation of a recent product to remove lipids and other matrix co-extractives in the analysis of pesticide residues and environmental contaminants in foods\". Journal of Chromatography A 182: 81\u20138. doi:10.1016\/j.foodchem.2015.02.141. PMID 25842312.   \n\n\u2191 Rajski, \u0141.; Lozano, A;. Ucl\u00e9s, A. et al. (2013). \"Determination of pesticide residues in high oil vegetal commodities by using various multi-residue methods and clean-ups followed by liquid chromatography tandem mass spectrometry\". Journal of Chromatography A 1304: 109\u201320. doi:10.1016\/j.chroma.2013.06.070. PMID 23871559.   \n\n\u2191 Lozano, A.; Rajski, \u0141.; Ucl\u00e9s, S. et al. (2014). \"Evaluation of zirconium dioxide-based sorbents to decrease the matrix effect in avocado and almond multiresidue pesticide analysis followed by gas chromatography tandem mass spectrometry\". Talanta 118: 68\u201383. doi:10.1016\/j.talanta.2013.09.053. PMID 24274272.   \n\n\u2191 Tuzimski, T.; Szubartowski, S. (2019). \"Method Development for Selected Bisphenols Analysis in Sweetened Condensed Milk from a Can and Breast Milk Samples by HPLC-DAD and HPLC-QqQ-MS: Comparison of Sorbents (Z-SEP, Z-SEP Plus, PSA, C18, Chitin and EMR-Lipid) for Clean-Up of QuEChERS Extract\". Molecules 24 (11): E2093. doi:10.3390\/molecules24112093. PMC PMC6600471. PMID 31159388. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6600471 .   \n\n\u2191 36.0 36.1 \"2002\/657\/EC: Commission Decision of 12 August 2002 implementing Council Directive 96\/23\/EC concerning the performance of analytical methods and the interpretation of results\". EUR-Lex. European Union. 17 August 2002. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32002D0657 .   \n\n\u2191 37.0 37.1 Han, Z.; Ren, Y.; Zhu, J. et al. (2012). \"Multianalysis of 35 mycotoxins in traditional Chinese medicines by ultra-high-performance liquid chromatography-tandem mass spectrometry coupled with accelerated solvent extraction\". Journal of Agricultural and Food Chemistry 60 (33): 8233-47. doi:10.1021\/jf301928r. PMID 22823451.   \n\n\u2191 Tournas, V.H.; Sapp, C.; Trucksess, M.W. (2012). \"Occurrence of aflatoxins in milk thistle herbal supplements\". Food Additives and Contaminated Part A. 29 (6): 994\u20139. doi:10.1080\/19440049.2012.664788. PMID 22439650.   \n\n\u2191 39.0 39.1 Arroyo-Manzanares, N.; Garc\u00eda-Campa\u00f1a, A.M.; G\u00e1miz-Gracia, L. (2013). \"Multiclass mycotoxin analysis in Silybum marianum by ultra high performance liquid chromatography-tandem mass spectrometry using a procedure based on QuEChERS and dispersive liquid-liquid microextraction\". Journal of Chromatography A. 1282: 11\u201319. doi:10.1016\/j.chroma.2013.01.072. PMID 23415469.   \n\n\u2191 Han, Z.; Ren, Y.; Zhou, H. et al. (2011). \"A rapid method for simultaneous determination of zearalenone, \u03b1-zearalenol, \u03b2-zearalenol, zearalanone, \u03b1-zearalanol and \u03b2-zearalanol in traditional Chinese medicines by ultra-high-performance liquid chromatography-tandem mass spectrometry\". Journal of Chromatography B. 879 (5\u20136): 411\u201320. doi:10.1016\/j.jchromb.2010.12.028. PMID 21242113.   \n\n\u2191 Smith, M.C.; Madec, S.; Coton, E. et al. (2016). \"Natural Co-Occurrence of Mycotoxins in Foods and Feeds and Their in vitro Combined Toxicological Effects\". Toxins 8 (4): 94. doi:10.3390\/toxins8040094. PMC PMC4848621. PMID 27023609. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4848621 .   \n\n\u2191 42.0 42.1 \"PPDB: Pesticide Properties DataBase\". University of Hertfordshire. 03 February 2007. http:\/\/sitem.herts.ac.uk\/aeru\/ppdb\/en\/ .   \n\n\u2191 \"Regulation (EC) No 1107\/2009 of the European Parliament and of the Council of 21 October 2009 concerning the placing of plant protection products on the market and repealing Council Directives 79\/117\/EEC and 91\/414\/EEC\". EUR-Lex. European Union. 24 November 2009. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=celex%3A32009R1107 .   \n\n\u2191 \"2004\/247\/EC: Commission Decision of 10 March 2004 concerning the non-inclusion of simazine in Annex I to Council Directive 91\/414\/EEC and the withdrawal of authorisations for plant protection products containing this active substance\". EUR-Lex. European Union. 16 March 2004. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX%3A32004D0247 .   \n\n\u2191 Castaldo, L.; Graziani, G.; Gaspari, A. et al. (2019). \"Target Analysis and Retrospective Screening of Multiple Mycotoxins in Pet Food Using UHPLC-Q-Orbitrap HRMS\". Toxins 11 (8): 411\u201320. doi:10.3390\/toxins11080434. PMC PMC6723864. PMID 31344880. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6723864 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added. The references are slightly out of order compared to the original, starting at reference 38, due to non-sequential ordering in the original Table 2.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\">https:\/\/www.limswiki.org\/index.php\/Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on cannabis researchLIMSwiki journal articles on cannabis testing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 10 March 2020, at 01:46.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 362 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","3eda7d7fb752e96c7816c5992924922e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Ultra-high-performance liquid chromatography coupled with quadrupole-Orbitrap high-resolution mass spectrometry for multi-residue analysis of mycotoxins and pesticides in botanical nutraceuticals<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabidiol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabidiol\" data-key=\"0c46162c3d7b370d7448646c25334265\">Cannabidiol<\/a> (CBD) food supplements made of <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_sativa\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis sativa\" data-key=\"618f17d4f6938557aacc017ee0f52bbd\">Cannabis sativa<\/a><\/i> L. extracts have quickly become popular products due to their health-promoting effects. However, potential <a href=\"https:\/\/en.wikipedia.org\/wiki\/Contamination\" class=\"extiw wiki-link\" title=\"wikipedia:Contamination\" data-key=\"9cbfd3981e731a2bcbbe30a9fc70826b\">contaminants<\/a>, such as <a href=\"https:\/\/en.wikipedia.org\/wiki\/Mycotoxin\" class=\"extiw wiki-link\" title=\"wikipedia:Mycotoxin\" data-key=\"c7cc1510382432c9c94b7339534860f8\">mycotoxins<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Pesticide\" class=\"extiw wiki-link\" title=\"wikipedia:Pesticide\" data-key=\"f8f48c48d08827872062ca688e326657\">pesticides<\/a>, can be coextracted during the manufacturing process and placed into the final product. Accordingly, a novel methodology using <a href=\"https:\/\/www.limswiki.org\/index.php\/High-performance_liquid_chromatography#Pump_pressure\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"b21062e39e1d314d1ff52e043a207816\">ultra-high-performance liquid chromatography<\/a> coupled with <a href=\"https:\/\/www.limswiki.org\/index.php\/Quadrupole\" title=\"Quadrupole\" class=\"wiki-link\" data-key=\"8455517e18fb352dd0ba87d4fbdaf6c6\">quadrupole<\/a>-<a href=\"https:\/\/www.limswiki.org\/index.php\/Orbitrap\" title=\"Orbitrap\" class=\"wiki-link\" data-key=\"19b0ec5804cc4b206e3c2d92cf9a8938\">Orbitrap<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/Resolution_(mass_spectrometry)\" title=\"Resolution (mass spectrometry)\" class=\"wiki-link\" data-key=\"8abe7f0b899e1e09b6805e197d0e890f\">high-resolution<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/Mass_spectrometry\" title=\"Mass spectrometry\" class=\"wiki-link\" data-key=\"fb548eafe2596c35d7ea741849aa83d4\">mass spectrometry<\/a> (UHPLC-Q-Orbitrap HRMS) was developed to quantify 16 mycotoxins produced by major <i>C. sativa<\/i> fungi, followed by a post-target screening of 283 pesticides based on a comprehensive spectral library. The validated procedure was applied to 10 CBD-based products. Up to six different <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Fusarium\" class=\"extiw wiki-link\" title=\"wikipedia:Fusarium\" data-key=\"d9a171381bde73b13a400f53c2668740\">Fusarium<\/a><\/i> mycotoxins were found in seven samples, the most prevalent being <a href=\"https:\/\/en.wikipedia.org\/wiki\/Zearalenone\" class=\"extiw wiki-link\" title=\"wikipedia:Zearalenone\" data-key=\"4b42e27fe606428023567dacc230a8ca\">zearalenone<\/a> (60%) and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Enniatin\" class=\"extiw wiki-link\" title=\"wikipedia:Enniatin\" data-key=\"ae99b64bbe346978e4887d430435124e\">enniatin B1<\/a> (30%), both found at a maximum level of 11.6 ng\/g. Co-occurrence was observed in four samples, including one with enniatin B1, enniatin A, and enniatin A1. On the other hand, 46 different pesticides were detected after retrospective analysis. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Ethoxyquin\" class=\"extiw wiki-link\" title=\"wikipedia:Ethoxyquin\" data-key=\"8bbae5de59fca9c7a3fc15b3989a103e\">Ethoxyquin<\/a> (50%), <a href=\"https:\/\/en.wikipedia.org\/wiki\/Piperonyl_butoxide\" class=\"extiw wiki-link\" title=\"wikipedia:Piperonyl butoxide\" data-key=\"74871923029ebcb4c82d15c09656a3c8\">piperonyl butoxide<\/a> (40%), <a href=\"https:\/\/en.wikipedia.org\/wiki\/Simazine\" class=\"extiw wiki-link\" title=\"wikipedia:Simazine\" data-key=\"1fe9465e2c37490fb4cda335a32c0766\">simazine<\/a> (30%), and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cyanazine\" class=\"extiw wiki-link\" title=\"wikipedia:Cyanazine\" data-key=\"b38d9b63124d1b822858cb0c0c91b81c\">cyanazine<\/a> (30%) were the major residues found. These results highlight the necessity of monitoring contaminants in food supplements in order to ensure safe consumption, even more considering the increasing trend in their use. Furthermore, the developed procedure is proposed as a powerful analytical tool to evaluate the potential mycotoxin profile of these particular products. To our knowledge, this represents the first multi-class analysis of CBD-based supplements regarding mycotoxins and pesticide residues using high-resolution mass spectrometry techniques.\n<\/p><p>Keywords: mycotoxins, pesticides, Q-Exactive Orbitrap, CBD capsule, nutraceutical\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Nutrition is known to be an essential component of quality state of health, and having an unbalanced diet can lead to several disorders and diseases.<sup id=\"rdp-ebb-cite_ref-AfshinHealth19_1-0\" class=\"reference\"><a href=\"#cite_note-AfshinHealth19-1\">[1]<\/a><\/sup> Due to current lifestyles, new, convenient ways to maintain proper dietary habits are required. Nutraceuticals have emerged as an alternative to increase the input of nutrients, contributing to an improvement in health. These products are bioactive compounds naturally occurring in food or produced <i>de novo<\/i> in human metabolism, biologicals, or botanicals, each intended to impart a physiological or medicinal effect after ingestion.<sup id=\"rdp-ebb-cite_ref-AlmadaNutra19_2-0\" class=\"reference\"><a href=\"#cite_note-AlmadaNutra19-2\">[2]<\/a><\/sup> They can be delivered either in foods and beverages or in other non-conventional forms, such as capsules, tablets, powders, or liquid extracts. In terms of marketing, nutraceuticals include a large number of different products packaged for specific groups by age, gender, physical condition, and activity level. The global market was valued at U.S. $109 billion in 2015 and was projected to reach U.S. $180 billion by 2020.<sup id=\"rdp-ebb-cite_ref-BinnsProb18_3-0\" class=\"reference\"><a href=\"#cite_note-BinnsProb18-3\">[3]<\/a><\/sup>\n<\/p><p>Inside the variety of products classified as nutraceuticals, food supplements based on botanical ingredients represent the second largest segment, behind vitamins and minerals. Most recently, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabidiol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabidiol\" data-key=\"0c46162c3d7b370d7448646c25334265\">cannabidiol<\/a> (CBD) dietary supplements made of <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_sativa\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis sativa\" data-key=\"618f17d4f6938557aacc017ee0f52bbd\">Cannabis sativa<\/a><\/i> L. extracts have quickly become popular products. CBD is a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabinoid\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabinoid\" data-key=\"fe5c4e73e0c21e16db393a214691296b\">phytocannabinoid<\/a> present in the resin secreted from <a href=\"https:\/\/en.wikipedia.org\/wiki\/Trichrome\" class=\"extiw wiki-link\" title=\"wikipedia:Trichrome\" data-key=\"4a5c3452359d3ad15f56e5c6455d4098\">trichomes<\/a> in female <i>C. sativa<\/i> plants and is mainly found in <a href=\"https:\/\/en.wikipedia.org\/wiki\/Inflorescence\" class=\"extiw wiki-link\" title=\"wikipedia:Inflorescence\" data-key=\"1308d1a34e9f736776cdaadb382e8917\">inflorescences<\/a>. The bioactivity of this compound has been related to an enhancement of its antioxidant and neurological activity, among others, by the promotion of several metabolic pathways.<sup id=\"rdp-ebb-cite_ref-ZuardiCanna08_4-0\" class=\"reference\"><a href=\"#cite_note-ZuardiCanna08-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CasaresCanna20_5-0\" class=\"reference\"><a href=\"#cite_note-CasaresCanna20-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MaroonReview18_6-0\" class=\"reference\"><a href=\"#cite_note-MaroonReview18-6\">[6]<\/a><\/sup> However, the European Union (E.U.) does not consider CBD supplements as a novel food<sup id=\"rdp-ebb-cite_ref-EURegulation228315_7-0\" class=\"reference\"><a href=\"#cite_note-EURegulation228315-7\">[7]<\/a><\/sup> and lets member states set their own rules over its marketing, leading to a convoluted situation in terms of regulation. Despite several ambiguities in its legislation, the European market for CBD-based supplements was valued at U.S. $318 million in 2018, with a strong growth projection.<sup id=\"rdp-ebb-cite_ref-BrightfieldIntern_8-0\" class=\"reference\"><a href=\"#cite_note-BrightfieldIntern-8\">[8]<\/a><\/sup>\n<\/p><p>Due to the complex nature of <i>C. sativa<\/i> and other botanicals, potential <a href=\"https:\/\/en.wikipedia.org\/wiki\/Contamination\" class=\"extiw wiki-link\" title=\"wikipedia:Contamination\" data-key=\"9cbfd3981e731a2bcbbe30a9fc70826b\">contaminants<\/a> can be coextracted during the different stages of the manufacturing process and placed into the final product. Among all the potential non-desirable compounds in herbal-based supplements, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Mycotoxin\" class=\"extiw wiki-link\" title=\"wikipedia:Mycotoxin\" data-key=\"c7cc1510382432c9c94b7339534860f8\">mycotoxins<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Pesticide\" class=\"extiw wiki-link\" title=\"wikipedia:Pesticide\" data-key=\"f8f48c48d08827872062ca688e326657\">pesticides<\/a> are the most commonly reported.<sup id=\"rdp-ebb-cite_ref-SantiniNutra18_9-0\" class=\"reference\"><a href=\"#cite_note-SantiniNutra18-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GulatiBotan19_10-0\" class=\"reference\"><a href=\"#cite_note-GulatiBotan19-10\">[10]<\/a><\/sup> Mycotoxins are secondary metabolites mainly produced by the fungi genera <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Fusarium\" class=\"extiw wiki-link\" title=\"wikipedia:Fusarium\" data-key=\"d9a171381bde73b13a400f53c2668740\">Fusarium<\/a><\/i>, <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Aspergillus\" class=\"extiw wiki-link\" title=\"wikipedia:Aspergillus\" data-key=\"8fa74c7e8a416c7d99056a355b2f0f3d\">Aspergillus<\/a><\/i>, <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Penicillium\" class=\"extiw wiki-link\" title=\"wikipedia:Penicillium\" data-key=\"1a4d0bd43b1413b7e44a67afb01bcf00\">Penicillium<\/a><\/i>, <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Claviceps\" class=\"extiw wiki-link\" title=\"wikipedia:Claviceps\" data-key=\"6361cdf8f3f216a42d38eb8a1b0b9b9d\">Claviceps<\/a><\/i>, and <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Alternaria\" class=\"extiw wiki-link\" title=\"wikipedia:Alternaria\" data-key=\"9e9751374c9a4cd8984a43968845bc43\">Alternaria<\/a><\/i>. These compounds can be present in food and feed commodities and display immunosuppressive, nephrotoxic, or carcinogenic effects, among others.<sup id=\"rdp-ebb-cite_ref-Rodr.C3.ADguez-CarrascoOccurr15_11-0\" class=\"reference\"><a href=\"#cite_note-Rodr.C3.ADguez-CarrascoOccurr15-11\">[11]<\/a><\/sup> According to their carcinogenic potential, some mycotoxins, like <a href=\"https:\/\/en.wikipedia.org\/wiki\/Aflotoxin\" class=\"extiw wiki-link\" title=\"wikipedia:Aflotoxin\" data-key=\"9755b10bc4bd379d2e91d5dd3ce80d26\">aflatoxins<\/a>, have been included in the classification list of human carcinogens provided by the International Agency for Research on Cancer (IARC).<sup id=\"rdp-ebb-cite_ref-OstryMycotox17_12-0\" class=\"reference\"><a href=\"#cite_note-OstryMycotox17-12\">[12]<\/a><\/sup> These mycotoxins are produced by the genera <i>Aspergillus<\/i>, which has been categorized as a major fungus occurring in <i>C. sativa<\/i> inflorescences alongside other mycotoxin producing fungi, like <i>Fusarium<\/i> spp., so different mycotoxins could be also expected.<sup id=\"rdp-ebb-cite_ref-McKernanCanna15_13-0\" class=\"reference\"><a href=\"#cite_note-McKernanCanna15-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McHardyInfect18_14-0\" class=\"reference\"><a href=\"#cite_note-McHardyInfect18-14\">[14]<\/a><\/sup> On the other hand, pesticides include a broad range of compounds routinely applied to protect crops from different pests. However, residues coming from these products can accumulate in plants intended for human consumption, leading to several health issues related to neurotoxicity, carcinogenicity, and pulmonotoxicity, as well as developmental and reproductive disorders.<sup id=\"rdp-ebb-cite_ref-AtapattuPest20_15-0\" class=\"reference\"><a href=\"#cite_note-AtapattuPest20-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChoudriPest19_16-0\" class=\"reference\"><a href=\"#cite_note-ChoudriPest19-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MostafalouPest17_17-0\" class=\"reference\"><a href=\"#cite_note-MostafalouPest17-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-YePest17_18-0\" class=\"reference\"><a href=\"#cite_note-YePest17-18\">[18]<\/a><\/sup>\n<\/p><p>In terms of regulation, maximum residue limits (MRL) for different types of contaminants have been set by the E.U. Regulation (EC) No. 396\/2005<sup id=\"rdp-ebb-cite_ref-EURegulation396_05_19-0\" class=\"reference\"><a href=\"#cite_note-EURegulation396_05-19\">[19]<\/a><\/sup> establishes limits for pesticides, whereas Regulation (EC) No. 1881\/2006<sup id=\"rdp-ebb-cite_ref-EURegulation1881_06_20-0\" class=\"reference\"><a href=\"#cite_note-EURegulation1881_06-20\">[20]<\/a><\/sup> covers mycotoxins, attaching maximum limits in food and feeds. Nevertheless, nutraceutical products are not considered by the legislation yet; however, due to a potential carryover during the manufacturing process, contamination could be expected not only in raw material, but also in other by-products. Moreover, several studies have reported the sole presence of pesticides<sup id=\"rdp-ebb-cite_ref-JeongPest08_21-0\" class=\"reference\"><a href=\"#cite_note-JeongPest08-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Gonz.C3.A1lez-Mart.C3.ADnPest18_22-0\" class=\"reference\"><a href=\"#cite_note-Gonz.C3.A1lez-Mart.C3.ADnPest18-22\">[22]<\/a><\/sup>, mycotoxins<sup id=\"rdp-ebb-cite_ref-VaclavikDeterm13_23-0\" class=\"reference\"><a href=\"#cite_note-VaclavikDeterm13-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-0\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup>, and both types of contaminants<sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15_25-0\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMulti16_26-0\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMulti16-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16_27-0\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16-27\">[27]<\/a><\/sup> in diverse food supplements, emphasizing the necessity to evaluate the contamination profile of these products considering their rising consumption and popularity.\n<\/p><p>To overcome this point, the development of analytical procedures is needed. Concerning the extraction of contaminants, <a href=\"https:\/\/www.limswiki.org\/index.php\/Quechers\" title=\"Quechers\" class=\"wiki-link\" data-key=\"d29caa5b19b907be733dffe4ee5b22a8\">QuEChERS<\/a> (quick, easy, cheap, effective, rugged and safe)<sup id=\"rdp-ebb-cite_ref-JeongPest08_21-1\" class=\"reference\"><a href=\"#cite_note-JeongPest08-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VaclavikDeterm13_23-1\" class=\"reference\"><a href=\"#cite_note-VaclavikDeterm13-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-1\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup> and \u201cdilute and shoot\u201d procedures have been recently applied to food supplements delivered as gelatin capsules, traditional capsules, tablets, powder extracts or liquid presentations.<sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15_25-1\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMulti16_26-1\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMulti16-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16_27-1\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16-27\">[27]<\/a><\/sup> Analytical methods used in the detection and quantification of contamination include <a href=\"https:\/\/www.limswiki.org\/index.php\/ELISA\" title=\"ELISA\" class=\"wiki-link\" data-key=\"f48f479ea5841a7362147598ac925bd6\">ELISA<\/a> detection<sup id=\"rdp-ebb-cite_ref-SantosScreen09_28-0\" class=\"reference\"><a href=\"#cite_note-SantosScreen09-28\">[28]<\/a><\/sup>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Gas_chromatography\" title=\"Gas chromatography\" class=\"wiki-link\" data-key=\"e621fc6f90266fbc8db27d516e9cbb94\">gas chromatography<\/a> (GC) coupled with <a href=\"https:\/\/www.limswiki.org\/index.php\/Mass_spectrometry\" title=\"Mass spectrometry\" class=\"wiki-link\" data-key=\"fb548eafe2596c35d7ea741849aa83d4\">mass spectrometry<\/a> (MS)<sup id=\"rdp-ebb-cite_ref-Gonz.C3.A1lez-Mart.C3.ADnPest18_22-1\" class=\"reference\"><a href=\"#cite_note-Gonz.C3.A1lez-Mart.C3.ADnPest18-22\">[22]<\/a><\/sup> and <a href=\"https:\/\/www.limswiki.org\/index.php\/High-performance_liquid_chromatography#Pump_pressure\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"b21062e39e1d314d1ff52e043a207816\">ultra-high-performance liquid chromatography<\/a> (UHPLC) coupled with <a href=\"https:\/\/www.limswiki.org\/index.php\/Tandem_mass_spectrometry\" title=\"Tandem mass spectrometry\" class=\"wiki-link\" data-key=\"55f167a11d8b5037392ba845986bf6bf\">tandem mass spectrometry<\/a> (MS\/MS)<sup id=\"rdp-ebb-cite_ref-VaclavikDeterm13_23-2\" class=\"reference\"><a href=\"#cite_note-VaclavikDeterm13-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-2\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup>, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Resolution_(mass_spectrometry)\" title=\"Resolution (mass spectrometry)\" class=\"wiki-link\" data-key=\"8abe7f0b899e1e09b6805e197d0e890f\">high-resolution<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/Orbitrap\" title=\"Orbitrap\" class=\"wiki-link\" data-key=\"19b0ec5804cc4b206e3c2d92cf9a8938\">Orbitrap<\/a> mass spectrometry (Q-Orbitrap HRMS).<sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15_25-2\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMulti16_26-2\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMulti16-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16_27-2\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16-27\">[27]<\/a><\/sup> Due to its high resolving power, sensitivity and accurate mass measurement, high-resolution mass spectrometry stands as a suitable alternative for evaluating a large number of contaminants present in complex matrices at low concentrations. Therefore, the aim of the present study was to provide an analysis of pesticide residues and mycotoxins produced by major <i>C. sativa<\/i> fungi occurring in CBD-based food supplements using ultra-high-performance liquid chromatography coupled with high-resolution Orbitrap mass spectrometry. To achieve this, a novel methodology was developed in order to identify and quantify 16 mycotoxins after evaluating different extraction procedures, followed by a post-target screening of 283 pesticides based on a comprehensive spectral library. To the best of the authors\u2019 knowledge, this is the first multi-class analysis of CBD-based supplements through the use of high-resolution mass spectrometry techniques.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Optimization_of_extraction_procedure\">Optimization of extraction procedure<\/span><\/h3>\n<p>The molecular complexity of this matrix demands an effective extraction in order to detect and quantify several mycotoxins in a reliable way. A QuEChERS methodology previously developed on this typology of sample<sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-3\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup> was selected as the starting point, whereas different volumes of extraction and the type of sorbent for clean-up was tested.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Evaluation_of_the_volume_of_extraction_solvent\">Evaluation of the volume of extraction solvent<\/span><\/h4>\n<p>The extraction procedure was first evaluated in triplicate by spiking the sample at 10 ng\/g using the following volumes of extraction solvent per gram of sample: 2.5, 5, 7.5 and 10 mL.\n<\/p><p>The extraction performed with 2.5 mL showed recovery values below the minimum limit (70%) for the vast majority of the studied analytes as a consequence of solvent saturation (Figure 1a). Satisfactory recoveries (70\u2013120%) were obtained after performing the extraction with 5 mL of solvent for the majority of compounds, with the exception of \u03b2-ZEL (155%) and ZAN (150%), which were significantly more efficient than the other volumes tested (p < 0.05). On the other hand, the extractions performed with 7.5 and 10 mL showed a gradual decrease in recoveries due to the larger dilution of the analytes. Therefore, 5 mL of AcN was selected as the optimal volume of extraction solvent for this type of CBD capsule.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Narv%C3%A1ez_Toxins2020_12-2.png\" class=\"image wiki-link\" data-key=\"a3fbbc9401199faf5ce9c54a4f717683\"><img alt=\"Fig1 Narv\u00e1ez Toxins2020 12-2.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2b\/Fig1_Narv%C3%A1ez_Toxins2020_12-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> Percentage of mycotoxins extracted with a recovery value (R) below 70% (white), between 70% and 120% (black) and above 120% (grey), corresponding to extractions performed with: (<b>a<\/b>) different volumes of solvent at a spiking level of 10 ng\/g; (<b>b<\/b>) different sorbents for clean-up at a spiking level of 10 ng\/g.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"Evaluation_of_the_type_of_sorbent_for_clean-up\">Evaluation of the type of sorbent for clean-up<\/span><\/h4>\n<p>The molecular composition of the soft gel capsules mainly consists on fatty acids and proteins. Because of the complex nature of this matrix, an efficient clean-up is required in order to avoid interference with the analytes. To achieve this, clean-up with different sorbents (100 mg) was performed, including octadecyl carbon chain-bonded silica (C18) (as previously suggested<sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-4\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup>), graphitized carbon black (GCB), primary-secondary amine (PSA), and zirconium oxide (Z-Sep+).\n<\/p><p>PSA exhibited a good performance for the vast majority of analytes (Figure 1b, above) but was unable to recover other important mycotoxins, such as AFB1 and AFG1. The moderate affinity of PSA with polar compounds may explain low recoveries for aflatoxins, being consistent with other works based on oily matrices.<sup id=\"rdp-ebb-cite_ref-ZhaoDeterm17_29-0\" class=\"reference\"><a href=\"#cite_note-ZhaoDeterm17-29\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Hidalgo-RuizARapid19_30-0\" class=\"reference\"><a href=\"#cite_note-Hidalgo-RuizARapid19-30\">[30]<\/a><\/sup> Similarly, extraction with C18 was efficient for most compounds, and only some low-polarity mycotoxins showed recoveries out of the range set, like ZAN (150%) and \u03b2-ZEL (155%). Clean-up using GCB showed poor results, allowing us to detect only NEO (85%), HT-2 (89%), and T-2 (89%). This sorbent is able to retain planar molecules; additionally, mycotoxin adsorption has been previously reported<sup id=\"rdp-ebb-cite_ref-MyresiotisDeterm15_31-0\" class=\"reference\"><a href=\"#cite_note-MyresiotisDeterm15-31\">[31]<\/a><\/sup>, which might be the reason for the low recoveries obtained here. Finally, extraction performed with Z-Sep+ showed satisfactory recoveries (70\u2013120%) for all the mycotoxins studied.\n<\/p><p>On the other hand, the influence of the matrix was minimal (80% \u2264 signal suppression\/enhancement (SSE) \u2264 120%) for all targeted analytes when using Z-Sep+ and PSA. Clean-up based on Z-Sep+ has been successfully applied to the extraction of analytes from lipid matrices.<sup id=\"rdp-ebb-cite_ref-HanEval16_32-0\" class=\"reference\"><a href=\"#cite_note-HanEval16-32\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RajskiDeterm13_33-0\" class=\"reference\"><a href=\"#cite_note-RajskiDeterm13-33\">[33]<\/a><\/sup> Furthermore, Z-Sep+ is also able to form irreversible links with carboxylic groups present in proteins<sup id=\"rdp-ebb-cite_ref-LozanoEval14_34-0\" class=\"reference\"><a href=\"#cite_note-LozanoEval14-34\">[34]<\/a><\/sup>, standing as the most suitable sorbent for the here-analyzed matrix. Similarly, the use of PSA has been suggested to remove coextracted fatty acids and other ionic lipids.<sup id=\"rdp-ebb-cite_ref-TuzimskiMethod19_35-0\" class=\"reference\"><a href=\"#cite_note-TuzimskiMethod19-35\">[35]<\/a><\/sup>\n<\/p><p>On the contrary, a strong matrix effect was evidenced for half the analytes when using C18 and GCB. Signal suppression was detected after using C18, obtaining SSE ranging from 40% to 69%, whereas signal enhancement occurred after GCB clean-up, with SSE increasing from 128% to 167%. Since both sorbents have a preferential affinity for non-polar compounds, matrix interferents were not fully removed but coextracted. The presence of these coextracted species can change the ionization efficiency, leading to improper SSE and an unreliable quantification. Although no significant differences were observed between the use of Z-Sep+ and PSA (<i>p<\/i> > 0.05), Z-Sep+ was chosen because of its better performance minimizing matrix interference.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Analytical_method_validation\">Analytical method validation<\/span><\/h3>\n<p>The optimized method was validated for the simultaneous extraction of 16 mycotoxins in CBD-based products. Results are shown in Table 1. Good linearity was observed for all analytes in the range assessed (0.20\u2013100 ng\/g), with regression coefficients (r2) above 0.990 and a deviation \u226420% for each level of the calibration curve. Comparison between calibration curves built in a blank matrix and in neat solvent showed a minimal interference in the matrix (\u00b120%) for the studied analytes. Hence, external calibration curves were used for quantification purposes. Limits of quality (LOQs) obtained for all studied analytes were between 0.20 and 6.25 ng\/g. Regarding trueness, recovery values corresponding to a fortification level of 20 ng\/g ranged between 63 and 103% and between 63 and 113% for the lowest fortification level (10 ng\/g). Referring to the additional spiking level (2 ng\/g) for aflatoxins, recoveries ranged between 63% and 86%. Precision study revealed both RSDr and RSDR values below 20% for all the mycotoxins analyzed. These results confirmed that the optimized procedure is suitable for a reliable quantification of the mycotoxins analyzed, fulfilling the criteria set by Commission Decision 2002\/657\/EC.<sup id=\"rdp-ebb-cite_ref-EURegulation657_02_36-0\" class=\"reference\"><a href=\"#cite_note-EURegulation657_02-36\">[36]<\/a><\/sup> \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Narv%C3%A1ez_Toxins2020_12-2.png\" class=\"image wiki-link\" data-key=\"8037dee14a1d52423934d34f81f931cb\"><img alt=\"Tab1 Narv\u00e1ez Toxins2020 12-2.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/07\/Tab1_Narv%C3%A1ez_Toxins2020_12-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 1<\/b> Method performance: linearity, matrix effect (SSE %), recovery and LOQ. (<sup>1<\/sup> Additional fortification level only for AFs)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Table 2 reviews the available literature regarding mycotoxins in herbal-based supplements. As shown, the here-obtained LOQs were lower than the ones reported in previous studies using UHPLC-Q-Orbitrap HRMS. As established by Regulation (EC) No. 1881\/2006<sup id=\"rdp-ebb-cite_ref-EURegulation1881_06_20-1\" class=\"reference\"><a href=\"#cite_note-EURegulation1881_06-20\">[20]<\/a><\/sup>, maximum limits for aflatoxins in many food matrices must not reach levels which are below those LOQs (5 ng\/g), whereas LOQs obtained in this study were between 5 and 25 times lower. Other analytical methods based on low resolution mass spectrometry<sup id=\"rdp-ebb-cite_ref-HanMultiOf12_37-0\" class=\"reference\"><a href=\"#cite_note-HanMultiOf12-37\">[37]<\/a><\/sup> required longer and more complicated extraction procedures than the QuEChERS developed here. Even ELISA detection has been used for quantification of mycotoxins in medicinal herbs<sup id=\"rdp-ebb-cite_ref-SantosScreen09_28-1\" class=\"reference\"><a href=\"#cite_note-SantosScreen09-28\">[28]<\/a><\/sup>, but a very specific extraction had to be performed for different groups of analytes using several multi-functional columns. The QuEChERS procedure developed in this study, in combination with UHPLC-Q-Orbitrap mass spectrometry, was extremely simple and reliable, allowing for the quantification of all mycotoxins with high sensitivity.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><b>Table 2.<\/b> Available methods for measurement of mycotoxins in herbal-based supplements <sup>1<\/sup><br \/> <br \/><sup>1<\/sup> ESI+ = positive ion mode; ESI\u2212 = negative ion mode; HRMS = high-resolution MS; LOQ = limit of quantification; MRM = multiple reaction monitoring; QQQ = triple quadrupole. <sup>2<\/sup> Range of LOQs referring to the analyzed mycotoxins. <sup>3<\/sup> AFs = aflatoxins; DON = deoxynivalenol; CIT = citrinin; FB1 = fumonisin B1; MPA = mycophenolic acid; OTA = ochratoxin A; OTB = ochratoxin B. NEO = neosolaniol; ZEN = zearalenone; ENNA, ENNA1, ENNB, and ENNB1 = enniatins.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Samples procedence (no.)\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Positives samples (%)\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Major analytes detected\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Concentration reported (ng\/g)\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\" colspan=\"3\">Determination\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Sensitivity (LOQ, ng\/g)\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Detection method\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Reference\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">Medicinal or aromatic herbs (84)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">99\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ZEN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0\u201344.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">ELISA detection (EIA reader, SIRIO S)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\"><sup id=\"rdp-ebb-cite_ref-SantosScreen09_28-2\" class=\"reference\"><a href=\"#cite_note-SantosScreen09-28\">[28]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">T-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\u2013256.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.28\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DON<sup>3<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20.5\u2013343.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CIT<sup>3<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14.9\u2013354.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">Traditional Chinese herbs (60)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">83\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ZEN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.1\u201315.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">QQQ (Applied Biosystems) ESI+ MRM mode\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\"><sup id=\"rdp-ebb-cite_ref-HanMultiOf12_37-1\" class=\"reference\"><a href=\"#cite_note-HanMultiOf12-37\">[37]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AFs<sup>3<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\u201319.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">MPA<sup>3<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\u201322.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Milk thistle (83)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AFB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04\u20131.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LC-FLD (Waters)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-TournasOccurr12_38-0\" class=\"reference\"><a href=\"#cite_note-TournasOccurr12-38\">[38]<\/a><\/sup>\n<\/td><\/tr>\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">Green coffee bean (50)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">36\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">OTA<sup>3<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\u2013136.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">QQQ (AB SCIEX) ESI+ and ESI- MRM mode\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\"><sup id=\"rdp-ebb-cite_ref-VaclavikDeterm13_23-3\" class=\"reference\"><a href=\"#cite_note-VaclavikDeterm13-23\">[23]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">OTB<sup>3<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\u201320.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">FB1<sup>3<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50\u2013415\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">MPA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\u2013395\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Milk thistle (7)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">T-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">363\u2013453.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">QQQ (AB SCIEX) ESI+ MRM mode\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\"><sup id=\"rdp-ebb-cite_ref-Arroyo-ManzanaresMulti13_39-0\" class=\"reference\"><a href=\"#cite_note-Arroyo-ManzanaresMulti13-39\">[39]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HT-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">826.9\u2013943.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">43.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"7\">Herbals (69)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"7\">96\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ZEN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\u2013824\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"7\">QQQ (AB SCIEX) ESI+ and ESI- MRM mode\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"7\"><sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-5\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">T-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">69\u20131870\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HT-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">59\u20131530\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ENNB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\u20139260\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ENNB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\u201310,900\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ENNA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\u20138340\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ENNA1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\u20132340\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\"><i>Ginkgo biloba<\/i> (8)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AFB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.0\u201354\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">Q-Orbitrap (Exactive, Thermo FisherScientific) ESI+ and ESI- HRMS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\"><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15_25-3\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15-25\">[25]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AFB2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\u2013300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">T-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18\u201320\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green tea (10)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">AFB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">5.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Q-Orbitrap (Exactive, Thermo FisherScientific) ESI+ and ESI- HRMS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\"><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMulti16_26-3\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMulti16-26\">[26]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Royal jelly (8)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Soy (11)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AFB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.2\u201317.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Q-Orbitrap (Exactive, Thermo FisherScientific) ESI+ and ESI- HRMS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\"><sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16_27-3\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16-27\">[27]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AFG2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\"><i>Cannabis sativa<\/i> (10)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">70\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ZEN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.2\u201311.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Q-Orbitrap (Exactive, Thermo FisherScientific) ESI+ and ESI- HRMS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Current study\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ENNB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><LOQ\u201311.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.56\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Application_to_commercial_CBD-based_products\">Application to commercial CBD-based products<\/span><\/h3>\n<p>The validated UHPLC-Q-Orbitrap HRMS procedure was applied to 10 commercially available samples in order to evaluate the occurrence of mycotoxins. Results are shown in Table 3. A considerable occurrence of mycotoxins was observed, since contamination with at least one analyte was found in 70% of the samples. Up to six different mycotoxins (T-2, ZAN, ZEN, ENNB1, ENNA, ENNA1) were quantified at a range from below LOQ to 11.6 ng\/g, all produced by <i>Fusarium<\/i> genera, reported as a major <i>C. sativa<\/i> pathogen fungus.<sup id=\"rdp-ebb-cite_ref-McHardyInfect18_14-1\" class=\"reference\"><a href=\"#cite_note-McHardyInfect18-14\">[14]<\/a><\/sup> Previous studies regarding mycotoxins in different herbal-based extracts have revealed the occurrence of similar mycotoxins independently of the matrix and the dosage form (Table 2, above). Despite the fact that the percentage of positive samples varied among the different studies (19\u201399%), when the sensitivity of the analytical method increased, reaching lower LOQs, the number of positive samples dramatically increased. This indicated that mycotoxin contamination in herbal-based products at low levels is frequent.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab3_Narv%C3%A1ez_Toxins2020_12-2.png\" class=\"image wiki-link\" data-key=\"c11ecb298c4ff9e75c496530efeb915b\"><img alt=\"Tab3 Narv\u00e1ez Toxins2020 12-2.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/28\/Tab3_Narv%C3%A1ez_Toxins2020_12-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 3<\/b> Occurrence of studied mycotoxins in the analyzed samples<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In the here-analyzed samples, ZEN appeared to be the most common mycotoxin, with an incidence of 60% and concentration levels ranging from 4.2 to 11.6 ng\/g (mean level = 6.9 ng\/g). A high incidence of ZEN has also been previously reported in supplements made of different herbals from Czech and U.S. retail markets (84%, <i>n<\/i> = 69) at a wide range of concentrations (5\u2013824 ng\/g, mean value = 75.7 ng\/g).<sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-6\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup> Moreover, ZEN was previously found in 96% of medicinal herbals from Spain (<i>n<\/i> = 84) as well, but in a tighter range (1\u201344.1 ng\/g, mean value = 8.9 ng\/g).<sup id=\"rdp-ebb-cite_ref-SantosScreen09_28-3\" class=\"reference\"><a href=\"#cite_note-SantosScreen09-28\">[28]<\/a><\/sup>\n<\/p><p>Referring to T-2, results reported contamination in one sample at 2.0 ng\/g, in contrast with the prevalent presence of T-2 in 78% (<i>n<\/i> = 69) of the same Czech and U.S. samples, at concentrations rising from 69 to 1,870 ng\/g (mean value = 162 ng\/g).<sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-7\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup> High levels of T-2 were also observed in milk thistle samples from Spain (363\u2013453.9, mean value = 408.9 ng\/g) in only two out of seven samples.<sup id=\"rdp-ebb-cite_ref-Arroyo-ManzanaresMulti13_39-1\" class=\"reference\"><a href=\"#cite_note-Arroyo-ManzanaresMulti13-39\">[39]<\/a><\/sup> In the other hand, T-2 was quantified in 98% (<i>n<\/i> = 84) of the Spanish medicinal herbals, but in much lower concentrations (0.6\u2013256 ng\/g, mean value = 22.645 ng\/g).<sup id=\"rdp-ebb-cite_ref-SantosScreen09_28-4\" class=\"reference\"><a href=\"#cite_note-SantosScreen09-28\">[28]<\/a><\/sup>\n<\/p><p>Similarly, ZAN was quantified in one sample at 1.9 ng\/g. This mycotoxin has been scarcely targeted in dietary supplement studies, but it has been previously quantified at similar concentrations as those here-reported in two samples of Chinese medicinal herbals (<i>n<\/i> = 33).<sup id=\"rdp-ebb-cite_ref-HanARapid11_40-0\" class=\"reference\"><a href=\"#cite_note-HanARapid11-40\">[40]<\/a><\/sup>\n<\/p><p>Results also showed ENN contamination. ENNB1, ENNA, and ENNA1 were found in the same sample at 11.6, 4.2 and 5.8 ng\/g, respectively, whereas ENNB1 was detected in two other samples below the LOQ (1.56 ng\/g). These emerging <i>Fusarium<\/i> mycotoxins have been previously found in herbal products (84\u201391%, <i>n<\/i> = 69) widely ranging from 5 ng\/g up to 10,900 ng\/g (mean value = 354 ng\/g).<sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-8\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup> Similarly, ENNB1 was the most common toxin out of these emerging <i>Fusarium<\/i> mycotoxins, being consistent with the results here obtained.\n<\/p><p>All the mycotoxins found in the present study correspond to low- to non-polar compounds, which should be prevalently expected due to the nature of the matrix.\n<\/p><p>Co-occurrence of at least two mycotoxins was also observed in four out of 10 samples. Results showed the presence of ZEN in combination with ENNs B1, A and A1, ZAN, or T-2, which are common associations found by previous studies in herbal-based supplements.<sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-9\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SantosScreen09_28-5\" class=\"reference\"><a href=\"#cite_note-SantosScreen09-28\">[28]<\/a><\/sup> It must be noted that synergic or additive effects have been observed as a consequence of these combinations in <i>in vitro<\/i> assays.<sup id=\"rdp-ebb-cite_ref-SmithNatural16_41-0\" class=\"reference\"><a href=\"#cite_note-SmithNatural16-41\">[41]<\/a><\/sup> Based on what has been discussed and considering the uprising trend of <i>C. sativa<\/i>-based products, alongside the use of environment-friendly raw materials cultivated without pesticides, quality controls regarding mycotoxins should be set for these products in order to ensure safe consumption.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Identification_of_non-target_compounds_through_retrospective_analysis_in_studied_samples\">Identification of non-target compounds through retrospective analysis in studied samples<\/span><\/h3>\n<p>The post-target screening approach allowed us to detect pesticide residues in the analyzed samples using a spectral library. Results are shown in Figure 2. Up to 46 different pesticides were tentatively identified based on the mass spectral library of the pesticides. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Ethoxyquin\" class=\"extiw wiki-link\" title=\"wikipedia:Ethoxyquin\" data-key=\"8bbae5de59fca9c7a3fc15b3989a103e\">Ethoxyquin<\/a> was putatively found in five samples, being the most prevalent pesticide. The main function of ethoxyquin is to avoid fungal contamination during the postharvest stage of the plant through its scald-preventive properties.<sup id=\"rdp-ebb-cite_ref-PPDBPest_42-0\" class=\"reference\"><a href=\"#cite_note-PPDBPest-42\">[42]<\/a><\/sup> Surprisingly, the use of this pesticide is forbidden by the European Commission's (EC) Decision 2011\/143\/EU. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Piperonyl_butoxide\" class=\"extiw wiki-link\" title=\"wikipedia:Piperonyl butoxide\" data-key=\"74871923029ebcb4c82d15c09656a3c8\">Piperonyl butoxide<\/a> was found in four samples. This compound is not a pesticide by itself but can inhibit the resistance mechanisms of insects, being widely used in combination with other different pesticides.<sup id=\"rdp-ebb-cite_ref-PPDBPest_42-1\" class=\"reference\"><a href=\"#cite_note-PPDBPest-42\">[42]<\/a><\/sup> The tentative presence of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Simazine\" class=\"extiw wiki-link\" title=\"wikipedia:Simazine\" data-key=\"1fe9465e2c37490fb4cda335a32c0766\">simazine<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cyanazine\" class=\"extiw wiki-link\" title=\"wikipedia:Cyanazine\" data-key=\"b38d9b63124d1b822858cb0c0c91b81c\">cyanazine<\/a>, both found in three different samples, must also be noted. The use of these pesticides was prohibited by EC Regulation No. 1107\/2009<sup id=\"rdp-ebb-cite_ref-EURegulation1107_09_43-0\" class=\"reference\"><a href=\"#cite_note-EURegulation1107_09-43\">[43]<\/a><\/sup> and Commission Decision 2004\/247\/EC<sup id=\"rdp-ebb-cite_ref-EURegulation247_04_44-0\" class=\"reference\"><a href=\"#cite_note-EURegulation247_04-44\">[44]<\/a><\/sup>, respectively. Therefore, the occurrence of forbidden pesticides found in the here-analyzed samples highlights the necessity of monitoring potential contaminants in <i>C. sativa<\/i>-derived products acquired from online shops.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Narv%C3%A1ez_Toxins2020_12-2.png\" class=\"image wiki-link\" data-key=\"86bfefac5381e731b4aeac73b27ba827\"><img alt=\"Fig2 Narv\u00e1ez Toxins2020 12-2.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b7\/Fig2_Narv%C3%A1ez_Toxins2020_12-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> Occurrence of non-target pesticides in analyzed samples after post-run retrospective screening.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>A sample preparation procedure based on a QuEChERS followed by UHPLC coupled with high-resolution Q-Orbitrap mass spectrometry was optimized in order to determine and quantify 16 mycotoxins in <i>C. sativa<\/i>-based supplements. The proposed methodology was validated following the E.U. criteria, ensuring a proper specificity, selectivity, linearity, trueness, and precision with a fast chromatography run performance (eight minutes). The validated procedure was applied to ten CBD-based supplements that are commercially available online, allowing us to quantify up to six different <i>Fusarium<\/i> mycotoxins in 70% of samples. ZEN was the most prevalent mycotoxin (60%) found at a maximum level of 11.6 ng\/g (mean value = 6.9 ng\/g). Co-occurrence was observed in four out of ten samples, including one sample with ENNB1, ENNA, and ENNA1. Additionally, a retrospective analysis of pesticide residues was performed. Up to 46 different pesticides were tentatively detected, including some forbidden in C. sativa cultivation. Considering the uprising trend of CBD-based products, quality controls regarding contaminants should be set for these products in order to ensure a safe consumption. Furthermore, the developed procedure is proposed as a powerful analytical tool to evaluate the potential mycotoxin profile of these particular products.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Chemicals_and_reagents\">Chemicals and reagents<\/span><\/h3>\n<p>Acetonitrile (AcN), methanol (MeOH), and water for LC mobile phase (HPLC grade) were acquired from Merck (Darmstadt, Germany). Formic acid and ammonium formate were obtained from Fluka (Milan, Italy). Sodium chloride (NaCl), magnesium sulfate (MgSO4), octadecyl carbon chain-bonded silica (C18), graphitized carbon black (GCB), primary-secondary amine (PSA), and zirconium oxide (Z-Sep+) were obtained from Sigma Aldrich (Milan, Italy).\n<\/p><p>Mycotoxin standards and metabolites, namely aflatoxins (AFB1, AFB2, AFG1, and AFG2), HT-2 toxin (HT-2), T-2 toxin (T-2), neosolaniol (NEO), zearalenone (ZEN), \u03b1-zearalenol (\u03b1-ZEL), \u03b2-zearalenol (\u03b2-ZEL), zearalanone (ZAN), beauvericin (BEA) and enniatins (ENNA, ENNA1, ENNB, and ENNB1) were purchased from Sigma Aldrich (Milan, Italy). Individual stock solutions of all analytes were prepared by diluting 1 mg of each mycotoxin in 1 mL of methanol. The working standard solution including all the mycotoxins was made by adequate diluting in MeOH:H2O (70:30 v\/v) 0.1% formic acid to reach the required concentrations for performing the spike experiments: 20, 10 and 2 \u00b5g\/mL. All solutions were kept in safe conditions at \u221220 \u00b0C.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sampling\">Sampling<\/span><\/h3>\n<p>For the analysis of real samples, 10 different CBD gelatin capsules were obtained from online shops based in different European countries. The capsules are made of gel mass, which contains gelatin, water, glycerin, and other minor additives, whereas the fill formulation consists of olive oil mixed with hemp oil containing CBD at certain concentrations. The weight of each capsule depended on the manufacturer; there were 0.25, 0.5, and 1 g capsules. Only soft gel capsules were studied since it was the prevalent presentation available for CBD supplements. On the other hand, one sample of CBD supplements delivered as soft gel capsules was acquired from a local store (Naples, Italy). After confirming the absence of contaminants, they were used for preparing fortified samples for recovery assays and matrix-matched standards for calibration purposes. All the samples were conserved in dark and cool conditions, as recommended by the manufacturer, until further analysis.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sample_preparation\">Sample preparation<\/span><\/h3>\n<p>The sample preparation procedure developed by Veprikova <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-VeprikovaMycotox15_24-10\" class=\"reference\"><a href=\"#cite_note-VeprikovaMycotox15-24\">[24]<\/a><\/sup> was selected as a starting point and then slightly modified, as follows: 1 g of sample was weighed into a 50 mL polytetrafluorethylene (PTFE) tube and mixed with 5 mL of 1% aqueous formic acid. The mixture was placed in an SKO-D XL orbital shaker (Argo Lab, Italy) for 30 minutes at 294 \u00d7 g. Then, 5 mL of AcN were added and the mixture was shaken for an additional 30 minutes at 294 \u00d7 g. After that, 0.5 g of sodium chloride and 2 g of magnesium sulfate were added and the tube was shaken for 1 minute by hand, followed by centrifugation at 4907 \u00d7 g for 15 minutes in an SL 16R centrifuge (Thermo Fisher Scientific LED GmbH, Germany). A 2 mL aliquot of the upper acetonitrile layer was taken for dispersive solid phase extraction (d-SPE) cleanup in a 15 mL PTFE tube containing 100 mg of Z-Sep+ sorbent and 300 mg of magnesium sulfate. The tube was vortexed for 1 minute and then centrifuged at 4907 \u00d7 g for 15 min. An aliquot of the supernatant (1 mL) was collected and filtered through a 0.2 \u00b5m PTFE filter (Phenomenex, Italy) into a vial prior to UHPLC-Q-Orbitrap HRMS analysis.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"UHPLC-Q-Orbitrap_HRMS_analysis\">UHPLC-Q-Orbitrap HRMS analysis<\/span><\/h3>\n<p>The qualitative and quantitative profiles of the mycotoxins were obtained using an ultra-high-pressure liquid chromatograph (UHPLC, Thermo Fisher Scientific, Waltham, MA, USA) equipped with a degassing system, a Dionex Ultimate 3000, a Quaternary UHPLC pump working at 1250 bar, an auto sampler device, and a thermostated (T = 30 \u00b0C) Luna Omega 1.6 \u00b5m (50 \u00d7 2.1 \u00b5m) column.\n<\/p><p>The eluent consisted of two different phases: A.) (H<sub>2<\/sub>O containing 0.1% formic acid and 5 mM ammonium formate) and B.) (MeOH containing 0.1% formic acid and 5 mM ammonium formate). The gradient elution for LC-Orbitrap HRMS analyses was applied as follows: an initial 0% of phase B was held for 1 minute, which linearly went up to 95% B over 1 minute and held for 0.5 minutes. Next, the gradient decreased to 75% B over 2.5 minutes and then decreased again to 60% B over 1 minute. Finally, the gradient turned to 0% B over 0.5 minutes and then the column was equilibrated for 1.5 minutes at 0% B. The total run time was 8 minutes, at a flow rate of 0.4 mL\/min. A total of 5 \u00b5L of the sample was injected. Detection was performed using a Q-Exactive mass spectrometer. The mass spectrometer was operated in both positive and negative ion mode using fast polarity switching by setting two scan events (full ion MS and all ion fragmentation (AIF)). Full scan data were acquired at a resolving power of 35,000 FWHM at <i>m\/z<\/i> 200.\n<\/p><p>The ion source parameters were: spray voltage 4 kV (-4 kV in ESI\u2212 mode); capillary temperature 290 \u00b0C; S-lens RF level 50; sheath gas pressure (N<sub>2<\/sub> > 95%) 35, auxiliary gas (N<sub>2<\/sub> > 95%) 10, and auxiliary gas heater temperature 305 \u00b0C. The value for automatic gain control (AGC) target was set at 1 \u00d7 10<sup>6<\/sup>, a scan range of <i>m\/z<\/i> 100 to 1000 was selected and the injection time was set to 200 ms. The scan rate was set at 2 scans\/s. For the scan event of AIF, the parameters in the positive and negative ion mode were: mass resolving power = 17,500 FWHM; maximum injection time = 200 ms; scan time = 0.10 s; ACG target = 1 \u00d7 10<sup>5<\/sup>; scan range = 100\u20131000 <i>m\/z<\/i>, isolation window to 5.0 <i>m\/z<\/i>, and retention time window to 30 seconds. The Orbitrap-MS parameters were optimized in a previous work.<sup id=\"rdp-ebb-cite_ref-CastaldoTarget19_45-0\" class=\"reference\"><a href=\"#cite_note-CastaldoTarget19-45\">[45]<\/a><\/sup> The exact mass for the studied compounds, including elemental composition, retention time (RT), theoretical masses and accurate mass errors for the detected ions are shown in Table 4. A mass error below 5 ppm, referring to the molecular ions, was set for identification. Retrospective screening was carried out on spectral data collected using a pesticide spectral library (Pesticide Spectral Library Version 1.1 for LibraryView\u2122 Software, AB SCIEX, Framingham, USA). For accurate mass measurement, identification and confirmation were performed at a mass tolerance of 5 ppm for the molecular ion and for both fragments at the intensity threshold of 1000. Data analysis and processing were performed using the Xcalibur software, v. 3.1.66.10.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab4_Narv%C3%A1ez_Toxins2020_12-2.png\" class=\"image wiki-link\" data-key=\"4e3801f09bd903bd8c111cf16ace2235\"><img alt=\"Tab4 Narv\u00e1ez Toxins2020 12-2.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/9a\/Tab4_Narv%C3%A1ez_Toxins2020_12-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 4.<\/b> Retention times, accurate mass and mass accuracy of mycotoxins evaluated<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Validation_parameters\">Validation parameters<\/span><\/h3>\n<p>An in-house validation study was conducted following the E.U. Commission Decision 2002\/657\/EC.<sup id=\"rdp-ebb-cite_ref-EURegulation657_02_36-1\" class=\"reference\"><a href=\"#cite_note-EURegulation657_02-36\">[36]<\/a><\/sup> The parameters evaluated were selectivity, specificity, linearity, trueness, repeatability (intra-day precision), within-reproducibility (inter-day precision), limit of quantification (LOQ), and limit of detection (LOD). The selectivity and specificity of the method were evaluated by analyzing both standard solutions and samples, comparing the retention time of the peaks corresponding to the analytes of interest alongside the determination of its precursor and product ion, with a mass error below 5 ppm. For linearity, standard solutions built in neat solvent and matrix-matched calibration were analyzed by spiking blank samples at eight concentration levels from 0.2 to 100 ng\/g. The slopes of each linear calibration function were compared in order to detect a signal suppression\/enhancement (SSE) effect due to the matrix interference. This effect was quantified following the equation: SSE (%) = matrix-matched calibration slope\/solvent calibration slope x 100. An SSE value of 100% was interpreted as no matrix interference in the concentration range evaluated. An SSE value above 100% revealed signal enhancement whereas a value below 100% indicated signal suppression. For trueness, recovery studies were evaluated by spiking three blank samples at three different levels. Additionally, a lower spike level was used only for aflatoxins. Intra-day precision (RSDr) was expressed as the relative standard deviation after three determinations in a single day (<i>n<\/i> = 3). Inter-day precision was calculated by repeating the measurements in triplicate on three non-consecutive days (<i>n<\/i> = 9) and expressed as relative standard deviation (RSDR). The LOD was defined as the minimum concentration where the molecular ion can be identified by the instrument (mass error value below 5 ppm) and the LOQ as the minimum concentration where a linear response (mass error value below 5 ppm) can be observed with an accuracy and precision of \u2264 20%.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Statistical_analysis\">Statistical analysis<\/span><\/h3>\n<p>Validation experiments were performed in triplicate and the results expressed as the average values alongside relative standard deviation (RSD, %). The Saphiro\u2013Wilk test was applied to evaluate normality and multivariant analysis was performed using a non-parametric Kruskal\u2013Wallis test, considering <i>p<\/i> values < 0.05 as significant. Analysis of data was carried out using IBM SPSS version 25 statistical software package (SPSS, Chicago, IL, USA).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors want to acknowledge Alice Valli for donating sample material.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Conceptualization, Y.R.-C. and A.N.; methodology, L.I. and A.N.; validation, L.C. and A.N.; formal analysis, A.N.; investigation, L.C., L.I. and A.N.; resources, A.R.; writing\u2014original draft preparation, A.N.; writing\u2014review and editing, Y.R.-C.; supervision, Y.R.-C. and A.R.; project administration, Y.R.-C. and A.R.; funding acquisition, A.R. All authors have read and agreed to the published version of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research received no external funding.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-AfshinHealth19-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AfshinHealth19_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Afshin, A.; Sur, P.J.; Fay, K.A. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6899507\" target=\"_blank\">\"Health effects of dietary risks in 195 countries, 1990-2017: A systematic analysis for the Global Burden of Disease Study 2017\"<\/a>. <i>Lancet<\/i> <b>393<\/b> (10184): 1958\u201372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0140-6736%2819%2930041-8\" target=\"_blank\">10.1016\/S0140-6736(19)30041-8<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6899507\/\" target=\"_blank\">PMC6899507<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30954305\" target=\"_blank\">30954305<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6899507\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6899507<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Health+effects+of+dietary+risks+in+195+countries%2C+1990-2017%3A+A+systematic+analysis+for+the+Global+Burden+of+Disease+Study+2017&rft.jtitle=Lancet&rft.aulast=Afshin%2C+A.%3B+Sur%2C+P.J.%3B+Fay%2C+K.A.+et+al.&rft.au=Afshin%2C+A.%3B+Sur%2C+P.J.%3B+Fay%2C+K.A.+et+al.&rft.date=2019&rft.volume=393&rft.issue=10184&rft.pages=1958%E2%80%9372&rft_id=info:doi\/10.1016%2FS0140-6736%2819%2930041-8&rft_id=info:pmc\/PMC6899507&rft_id=info:pmid\/30954305&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6899507&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlmadaNutra19-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlmadaNutra19_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Almada, A.L. (2019). \"Chapter 1: Nutraceuticals and functional foods: Innovation, insulation, evangelism, and evidence\". In Bagchi, D.. <i>Nutraceutical and Functional Food Regulations in the United States and around the World<\/i> (3rd ed.). Academic Press. pp. 3\u201311. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780128164679.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+1%3A+Nutraceuticals+and+functional+foods%3A+Innovation%2C+insulation%2C+evangelism%2C+and+evidence&rft.atitle=Nutraceutical+and+Functional+Food+Regulations+in+the+United+States+and+around+the+World&rft.aulast=Almada%2C+A.L.&rft.au=Almada%2C+A.L.&rft.date=2019&rft.pages=pp.%26nbsp%3B3%E2%80%9311&rft.edition=3rd&rft.pub=Academic+Press&rft.isbn=9780128164679&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BinnsProb18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BinnsProb18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Binns, C.W.; Lee, M.K.; Lee, A.H. (2018). \"Problems and Prospects: Public Health Regulation of Dietary Supplements\". <i>Annual Review of Public Health<\/i> <b>39<\/b>: 403\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1146%2Fannurev-publhealth-040617-013638\" target=\"_blank\">10.1146\/annurev-publhealth-040617-013638<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29272167\" target=\"_blank\">29272167<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Problems+and+Prospects%3A+Public+Health+Regulation+of+Dietary+Supplements&rft.jtitle=Annual+Review+of+Public+Health&rft.aulast=Binns%2C+C.W.%3B+Lee%2C+M.K.%3B+Lee%2C+A.H.&rft.au=Binns%2C+C.W.%3B+Lee%2C+M.K.%3B+Lee%2C+A.H.&rft.date=2018&rft.volume=39&rft.pages=403%E2%80%9320&rft_id=info:doi\/10.1146%2Fannurev-publhealth-040617-013638&rft_id=info:pmid\/29272167&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZuardiCanna08-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZuardiCanna08_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zuardi, A.W. (2008). \"Cannabidiol: From an inactive cannabinoid to a drug with wide spectrum of action\". <i>Brazilian Journal of Psychiatry<\/i> <b>30<\/b> (3): 271\u201380. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1590%2Fs1516-44462008000300015\" target=\"_blank\">10.1590\/s1516-44462008000300015<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18833429\" target=\"_blank\">18833429<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabidiol%3A+From+an+inactive+cannabinoid+to+a+drug+with+wide+spectrum+of+action&rft.jtitle=Brazilian+Journal+of+Psychiatry&rft.aulast=Zuardi%2C+A.W.&rft.au=Zuardi%2C+A.W.&rft.date=2008&rft.volume=30&rft.issue=3&rft.pages=271%E2%80%9380&rft_id=info:doi\/10.1590%2Fs1516-44462008000300015&rft_id=info:pmid\/18833429&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CasaresCanna20-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CasaresCanna20_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Casares, L.; Garc\u00eda, V.; Garrido-Rodr\u00edguez, M. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6742916\" target=\"_blank\">\"Cannabidiol induces antioxidant pathways in keratinocytes by targeting BACH1\"<\/a>. <i>Redox Biology<\/i> <b>28<\/b>: 101321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.redox.2019.101321\" target=\"_blank\">10.1016\/j.redox.2019.101321<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6742916\/\" target=\"_blank\">PMC6742916<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31518892\" target=\"_blank\">31518892<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6742916\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6742916<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabidiol+induces+antioxidant+pathways+in+keratinocytes+by+targeting+BACH1&rft.jtitle=Redox+Biology&rft.aulast=Casares%2C+L.%3B+Garc%C3%ADa%2C+V.%3B+Garrido-Rodr%C3%ADguez%2C+M.+et+al.&rft.au=Casares%2C+L.%3B+Garc%C3%ADa%2C+V.%3B+Garrido-Rodr%C3%ADguez%2C+M.+et+al.&rft.date=2020&rft.volume=28&rft.pages=101321&rft_id=info:doi\/10.1016%2Fj.redox.2019.101321&rft_id=info:pmc\/PMC6742916&rft_id=info:pmid\/31518892&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6742916&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaroonReview18-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaroonReview18_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maroon, J.; Bost, J. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5938896\" target=\"_blank\">\"Review of the neurological benefits of phytocannabinoids\"<\/a>. <i>Surgical Neurology International<\/i> <b>9<\/b>: 91. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2Fsni.sni_45_18\" target=\"_blank\">10.4103\/sni.sni_45_18<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5938896\/\" target=\"_blank\">PMC5938896<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29770251\" target=\"_blank\">29770251<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5938896\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5938896<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Review+of+the+neurological+benefits+of+phytocannabinoids&rft.jtitle=Surgical+Neurology+International&rft.aulast=Maroon%2C+J.%3B+Bost%2C+J.&rft.au=Maroon%2C+J.%3B+Bost%2C+J.&rft.date=2018&rft.volume=9&rft.pages=91&rft_id=info:doi\/10.4103%2Fsni.sni_45_18&rft_id=info:pmc\/PMC5938896&rft_id=info:pmid\/29770251&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5938896&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EURegulation228315-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EURegulation228315_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/en\/TXT\/?uri=CELEX%3A32015R2283\" target=\"_blank\">\"Regulation (EU) 2015\/2283 of the European Parliament and of the Council of 25 November 2015 on novel foods, amending Regulation (EU) No 1169\/2011 of the European Parliament and of the Council and repealing Regulation (EC) No 258\/97 of the European Parliament and of the Council and Commission Regulation (EC) No 1852\/2001\"<\/a>. <i>EUR-Lex<\/i>. European Union. 11 December 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/en\/TXT\/?uri=CELEX%3A32015R2283\" target=\"_blank\">https:\/\/eur-lex.europa.eu\/legal-content\/en\/TXT\/?uri=CELEX%3A32015R2283<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Regulation+%28EU%29+2015%2F2283+of+the+European+Parliament+and+of+the+Council+of+25+November+2015+on+novel+foods%2C+amending+Regulation+%28EU%29+No+1169%2F2011+of+the+European+Parliament+and+of+the+Council+and+repealing+Regulation+%28EC%29+No+258%2F97+of+the+European+Parliament+and+of+the+Council+and+Commission+Regulation+%28EC%29+No+1852%2F2001&rft.atitle=EUR-Lex&rft.date=11+December+2015&rft.pub=European+Union&rft_id=https%3A%2F%2Feur-lex.europa.eu%2Flegal-content%2Fen%2FTXT%2F%3Furi%3DCELEX%253A32015R2283&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrightfieldIntern-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrightfieldIntern_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.brightfieldgroup.com\/international-cbd-cannabis-market-landscape\" target=\"_blank\">\"Iternational CBD and Cannabis Market Landscape\"<\/a>. <i>Brightfield Group Data Portals<\/i>. Brightfield Group<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.brightfieldgroup.com\/international-cbd-cannabis-market-landscape\" target=\"_blank\">https:\/\/www.brightfieldgroup.com\/international-cbd-cannabis-market-landscape<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Iternational+CBD+and+Cannabis+Market+Landscape&rft.atitle=Brightfield+Group+Data+Portals&rft.pub=Brightfield+Group&rft_id=https%3A%2F%2Fwww.brightfieldgroup.com%2Finternational-cbd-cannabis-market-landscape&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SantiniNutra18-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SantiniNutra18_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Santini, A.; Cammarata, S.M.; Capone, G. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5867125\" target=\"_blank\">\"Nutraceuticals: Opening the debate for a regulatory framework\"<\/a>. <i>British Journal of Clinical Pharmacology<\/i> <b>84<\/b> (4): 659\u2013672. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fbcp.13496\" target=\"_blank\">10.1111\/bcp.13496<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5867125\/\" target=\"_blank\">PMC5867125<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29433155\" target=\"_blank\">29433155<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5867125\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5867125<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Nutraceuticals%3A+Opening+the+debate+for+a+regulatory+framework&rft.jtitle=British+Journal+of+Clinical+Pharmacology&rft.aulast=Santini%2C+A.%3B+Cammarata%2C+S.M.%3B+Capone%2C+G.+et+al.&rft.au=Santini%2C+A.%3B+Cammarata%2C+S.M.%3B+Capone%2C+G.+et+al.&rft.date=2018&rft.volume=84&rft.issue=4&rft.pages=659%E2%80%93672&rft_id=info:doi\/10.1111%2Fbcp.13496&rft_id=info:pmc\/PMC5867125&rft_id=info:pmid\/29433155&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5867125&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GulatiBotan19-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GulatiBotan19_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Gulati, O.P.; Ottaway, P.B.; Jenning, S. et al. (2019). \"Chapter 20: Botanical nutraceuticals (food supplements and fortified and functional foods) and novel foods in the EU, with a main focus on legislative controls on safety aspects\". In Bagchi, D.. <i>Nutraceutical and Functional Food Regulations in the United States and around the World<\/i> (3rd ed.). Academic Press. pp. 277\u2013321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780128164679.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+20%3A+Botanical+nutraceuticals+%28food+supplements+and+fortified+and+functional+foods%29+and+novel+foods+in+the+EU%2C+with+a+main+focus+on+legislative+controls+on+safety+aspects&rft.atitle=Nutraceutical+and+Functional+Food+Regulations+in+the+United+States+and+around+the+World&rft.aulast=Gulati%2C+O.P.%3B+Ottaway%2C+P.B.%3B+Jenning%2C+S.+et+al.&rft.au=Gulati%2C+O.P.%3B+Ottaway%2C+P.B.%3B+Jenning%2C+S.+et+al.&rft.date=2019&rft.pages=pp.%26nbsp%3B277%E2%80%93321&rft.edition=3rd&rft.pub=Academic+Press&rft.isbn=9780128164679&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Rodr.C3.ADguez-CarrascoOccurr15-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Rodr.C3.ADguez-CarrascoOccurr15_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rodr\u00edguez-Carrasco, Y.; Fattore, M.; Albrizio, S. et al. (2015). \"Occurrence of Fusarium mycotoxins and their dietary intake through beer consumption by the European population\". <i>Food Chemistry<\/i> <b>178<\/b>: 149\u201355. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.foodchem.2015.01.092\" target=\"_blank\">10.1016\/j.foodchem.2015.01.092<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25704695\" target=\"_blank\">25704695<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Occurrence+of+Fusarium+mycotoxins+and+their+dietary+intake+through+beer+consumption+by+the+European+population&rft.jtitle=Food+Chemistry&rft.aulast=Rodr%C3%ADguez-Carrasco%2C+Y.%3B+Fattore%2C+M.%3B+Albrizio%2C+S.+et+al.&rft.au=Rodr%C3%ADguez-Carrasco%2C+Y.%3B+Fattore%2C+M.%3B+Albrizio%2C+S.+et+al.&rft.date=2015&rft.volume=178&rft.pages=149%E2%80%9355&rft_id=info:doi\/10.1016%2Fj.foodchem.2015.01.092&rft_id=info:pmid\/25704695&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OstryMycotox17-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OstryMycotox17_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ostry, V.; Malir, F.; Toman, J. et al. (2017). \"Mycotoxins as human carcinogens-the IARC Monographs classification\". <i>Mycotoxin Research<\/i> <b>33<\/b> (1): 65\u201373. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs12550-016-0265-7\" target=\"_blank\">10.1007\/s12550-016-0265-7<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27888487\" target=\"_blank\">27888487<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Mycotoxins+as+human+carcinogens-the+IARC+Monographs+classification&rft.jtitle=Mycotoxin+Research&rft.aulast=Ostry%2C+V.%3B+Malir%2C+F.%3B+Toman%2C+J.+et+al.&rft.au=Ostry%2C+V.%3B+Malir%2C+F.%3B+Toman%2C+J.+et+al.&rft.date=2017&rft.volume=33&rft.issue=1&rft.pages=65%E2%80%9373&rft_id=info:doi\/10.1007%2Fs12550-016-0265-7&rft_id=info:pmid\/27888487&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McKernanCanna15-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McKernanCanna15_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McKernan, K.; Spangler, J.; Zhang, L. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4897766\" target=\"_blank\">\"Cannabis microbiome sequencing reveals several mycotoxic fungi native to dispensary grade Cannabis flowers\"<\/a>. <i>F1000Research<\/i> <b>4<\/b>: 1422. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.12688%2Ff1000research.7507.2\" target=\"_blank\">10.12688\/f1000research.7507.2<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4897766\/\" target=\"_blank\">PMC4897766<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27303623\" target=\"_blank\">27303623<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4897766\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4897766<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis+microbiome+sequencing+reveals+several+mycotoxic+fungi+native+to+dispensary+grade+Cannabis+flowers&rft.jtitle=F1000Research&rft.aulast=McKernan%2C+K.%3B+Spangler%2C+J.%3B+Zhang%2C+L.+et+al.&rft.au=McKernan%2C+K.%3B+Spangler%2C+J.%3B+Zhang%2C+L.+et+al.&rft.date=2015&rft.volume=4&rft.pages=1422&rft_id=info:doi\/10.12688%2Ff1000research.7507.2&rft_id=info:pmc\/PMC4897766&rft_id=info:pmid\/27303623&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4897766&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McHardyInfect18-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-McHardyInfect18_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-McHardyInfect18_14-1\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McHardy, I.; Romanelli, A.; Harris, L.J. et al. (2018). \"Infectious risks associated with medicinal Cannabis: Potential implications for immunocompromised patients?\". <i>Journal of Infection<\/i> <b>76<\/b> (5): 500\u20131. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jinf.2018.01.010\" target=\"_blank\">10.1016\/j.jinf.2018.01.010<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29408325\" target=\"_blank\">29408325<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Infectious+risks+associated+with+medicinal+Cannabis%3A+Potential+implications+for+immunocompromised+patients%3F&rft.jtitle=Journal+of+Infection&rft.aulast=McHardy%2C+I.%3B+Romanelli%2C+A.%3B+Harris%2C+L.J.+et+al.&rft.au=McHardy%2C+I.%3B+Romanelli%2C+A.%3B+Harris%2C+L.J.+et+al.&rft.date=2018&rft.volume=76&rft.issue=5&rft.pages=500%E2%80%931&rft_id=info:doi\/10.1016%2Fj.jinf.2018.01.010&rft_id=info:pmid\/29408325&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AtapattuPest20-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AtapattuPest20_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Atapattu, S.N.; Johnson, K.R.D. (2020). \"Pesticide analysis in cannabis products\". <i>Journal of Chromatography A.<\/i> <b>1612<\/b>: 460656. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.chroma.2019.460656\" target=\"_blank\">10.1016\/j.chroma.2019.460656<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31679712\" target=\"_blank\">31679712<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pesticide+analysis+in+cannabis+products&rft.jtitle=Journal+of+Chromatography+A.&rft.aulast=Atapattu%2C+S.N.%3B+Johnson%2C+K.R.D.&rft.au=Atapattu%2C+S.N.%3B+Johnson%2C+K.R.D.&rft.date=2020&rft.volume=1612&rft.pages=460656&rft_id=info:doi\/10.1016%2Fj.chroma.2019.460656&rft_id=info:pmid\/31679712&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChoudriPest19-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChoudriPest19_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Choudri, B.S.; Charabi, Y. (2019). \"Pesticides and herbicides\". <i>Water Environment Research<\/i> <b>91<\/b> (10): 1342-1349. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fwer.1227\" target=\"_blank\">10.1002\/wer.1227<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31523896\" target=\"_blank\">31523896<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pesticides+and+herbicides&rft.jtitle=Water+Environment+Research&rft.aulast=Choudri%2C+B.S.%3B+Charabi%2C+Y.&rft.au=Choudri%2C+B.S.%3B+Charabi%2C+Y.&rft.date=2019&rft.volume=91&rft.issue=10&rft.pages=1342-1349&rft_id=info:doi\/10.1002%2Fwer.1227&rft_id=info:pmid\/31523896&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MostafalouPest17-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MostafalouPest17_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mostafalou, S.; Abdollahi, M. (2017). \"Pesticides: An update of human exposure and toxicity\". <i>Archives of Toxicology<\/i> <b>91<\/b> (2): 549-599. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00204-016-1849-x\" target=\"_blank\">10.1007\/s00204-016-1849-x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27722929\" target=\"_blank\">27722929<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pesticides%3A+An+update+of+human+exposure+and+toxicity&rft.jtitle=Archives+of+Toxicology&rft.aulast=Mostafalou%2C+S.%3B+Abdollahi%2C+M.&rft.au=Mostafalou%2C+S.%3B+Abdollahi%2C+M.&rft.date=2017&rft.volume=91&rft.issue=2&rft.pages=549-599&rft_id=info:doi\/10.1007%2Fs00204-016-1849-x&rft_id=info:pmid\/27722929&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YePest17-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YePest17_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ye, M.; Beach, J.; Martin, J.W. et al. (2017). \"Pesticide exposures and respiratory health in general populations\". <i>Journal of Environmental Sciences<\/i> <b>51<\/b>: 361\u201370. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jes.2016.11.012\" target=\"_blank\">10.1016\/j.jes.2016.11.012<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28115149\" target=\"_blank\">28115149<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pesticide+exposures+and+respiratory+health+in+general+populations&rft.jtitle=Journal+of+Environmental+Sciences&rft.aulast=Ye%2C+M.%3B+Beach%2C+J.%3B+Martin%2C+J.W.+et+al.&rft.au=Ye%2C+M.%3B+Beach%2C+J.%3B+Martin%2C+J.W.+et+al.&rft.date=2017&rft.volume=51&rft.pages=361%E2%80%9370&rft_id=info:doi\/10.1016%2Fj.jes.2016.11.012&rft_id=info:pmid\/28115149&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EURegulation396_05-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EURegulation396_05_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32005R0396\" target=\"_blank\">\"Regulation (EC) No 396\/2005 of the European Parliament and of the Council of 23 February 2005 on maximum residue levels of pesticides in or on food and feed of plant and animal origin and amending Council Directive 91\/414\/EECText with EEA relevance\"<\/a>. <i>EUR-Lex<\/i>. European Union. 16 March 2005<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32005R0396\" target=\"_blank\">https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32005R0396<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Regulation+%28EC%29+No+396%2F2005+of+the+European+Parliament+and+of+the+Council+of+23+February+2005+on+maximum+residue+levels+of+pesticides+in+or+on+food+and+feed+of+plant+and+animal+origin+and+amending+Council+Directive+91%2F414%2FEECText+with+EEA+relevance&rft.atitle=EUR-Lex&rft.date=16+March+2005&rft.pub=European+Union&rft_id=https%3A%2F%2Feur-lex.europa.eu%2Flegal-content%2FEN%2FALL%2F%3Furi%3DCELEX%253A32005R0396&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EURegulation1881_06-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-EURegulation1881_06_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-EURegulation1881_06_20-1\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32006R1881\" target=\"_blank\">\"Commission Regulation (EC) No 1881\/2006 of 19 December 2006 setting maximum levels for certain contaminants in foodstuffs\"<\/a>. <i>EUR-Lex<\/i>. European Union. 20 December 2006<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32006R1881\" target=\"_blank\">https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32006R1881<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Commission+Regulation+%28EC%29+No+1881%2F2006+of+19+December+2006+setting+maximum+levels+for+certain+contaminants+in+foodstuffs&rft.atitle=EUR-Lex&rft.date=20+December+2006&rft.pub=European+Union&rft_id=https%3A%2F%2Feur-lex.europa.eu%2Flegal-content%2FEN%2FALL%2F%3Furi%3DCELEX%253A32006R1881&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JeongPest08-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JeongPest08_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-JeongPest08_21-1\">21.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jeong, M.L.; Zahn, M.; Trinh, T. et al. (2008). \"Pesticide residue analysis of a dietary ingredient by gas chromatography\/selected-ion monitoring mass spectrometry using neutral alumina solid-phase extraction cleanup\". <i>Journal of AOAC International<\/i> <b>91<\/b> (3): 630-6. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18567310\" target=\"_blank\">18567310<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pesticide+residue+analysis+of+a+dietary+ingredient+by+gas+chromatography%2Fselected-ion+monitoring+mass+spectrometry+using+neutral+alumina+solid-phase+extraction+cleanup&rft.jtitle=Journal+of+AOAC+International&rft.aulast=Jeong%2C+M.L.%3B+Zahn%2C+M.%3B+Trinh%2C+T.+et+al.&rft.au=Jeong%2C+M.L.%3B+Zahn%2C+M.%3B+Trinh%2C+T.+et+al.&rft.date=2008&rft.volume=91&rft.issue=3&rft.pages=630-6&rft_id=info:pmid\/18567310&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Gonz.C3.A1lez-Mart.C3.ADnPest18-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Gonz.C3.A1lez-Mart.C3.ADnPest18_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-Gonz.C3.A1lez-Mart.C3.ADnPest18_22-1\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gonz\u00e1lez-Mart\u00edn, M.I.; Revilla, I.; Betances-Salcedo, E.V. et al. (2018). \"Pesticide residues and heavy metals in commercially processed propolis\". <i>Microchemical Journal<\/i> <b>143<\/b>: 423\u201329. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.microc.2018.08.040\" target=\"_blank\">10.1016\/j.microc.2018.08.040<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pesticide+residues+and+heavy+metals+in+commercially+processed+propolis&rft.jtitle=Microchemical+Journal&rft.aulast=Gonz%C3%A1lez-Mart%C3%ADn%2C+M.I.%3B+Revilla%2C+I.%3B+Betances-Salcedo%2C+E.V.+et+al.&rft.au=Gonz%C3%A1lez-Mart%C3%ADn%2C+M.I.%3B+Revilla%2C+I.%3B+Betances-Salcedo%2C+E.V.+et+al.&rft.date=2018&rft.volume=143&rft.pages=423%E2%80%9329&rft_id=info:doi\/10.1016%2Fj.microc.2018.08.040&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VaclavikDeterm13-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VaclavikDeterm13_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-VaclavikDeterm13_23-1\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-VaclavikDeterm13_23-2\">23.2<\/a><\/sup> <sup><a href=\"#cite_ref-VaclavikDeterm13_23-3\">23.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vaclavik, L.; Vaclavikova, M.; Begley, T.H. et al. (2013). \"Determination of multiple mycotoxins in dietary supplements containing green coffee bean extracts using ultrahigh-performance liquid chromatography-tandem mass spectrometry (UHPLC-MS\/MS)\". <i>Journal of Agricultural and Food Chemistry<\/i> <b>61<\/b> (20): 4822-30. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fjf401139u\" target=\"_blank\">10.1021\/jf401139u<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23631685\" target=\"_blank\">23631685<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determination+of+multiple+mycotoxins+in+dietary+supplements+containing+green+coffee+bean+extracts+using+ultrahigh-performance+liquid+chromatography-tandem+mass+spectrometry+%28UHPLC-MS%2FMS%29&rft.jtitle=Journal+of+Agricultural+and+Food+Chemistry&rft.aulast=Vaclavik%2C+L.%3B+Vaclavikova%2C+M.%3B+Begley%2C+T.H.+et+al.&rft.au=Vaclavik%2C+L.%3B+Vaclavikova%2C+M.%3B+Begley%2C+T.H.+et+al.&rft.date=2013&rft.volume=61&rft.issue=20&rft.pages=4822-30&rft_id=info:doi\/10.1021%2Fjf401139u&rft_id=info:pmid\/23631685&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VeprikovaMycotox15-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-0\">24.00<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-1\">24.01<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-2\">24.02<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-3\">24.03<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-4\">24.04<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-5\">24.05<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-6\">24.06<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-7\">24.07<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-8\">24.08<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-9\">24.09<\/a><\/sup> <sup><a href=\"#cite_ref-VeprikovaMycotox15_24-10\">24.10<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Veprikova, Z.; Zachariasova, M;. Dzuman, Z. et al. (2015). \"Mycotoxins in Plant-Based Dietary Supplements: Hidden Health Risk for Consumers\". <i>Journal of Agricultural and Food Chemistry<\/i> <b>63<\/b> (29): 6633-43. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Facs.jafc.5b02105\" target=\"_blank\">10.1021\/acs.jafc.5b02105<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26168136\" target=\"_blank\">26168136<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Mycotoxins+in+Plant-Based+Dietary+Supplements%3A+Hidden+Health+Risk+for+Consumers&rft.jtitle=Journal+of+Agricultural+and+Food+Chemistry&rft.aulast=Veprikova%2C+Z.%3B+Zachariasova%2C+M%3B.+Dzuman%2C+Z.+et+al.&rft.au=Veprikova%2C+Z.%3B+Zachariasova%2C+M%3B.+Dzuman%2C+Z.+et+al.&rft.date=2015&rft.volume=63&rft.issue=29&rft.pages=6633-43&rft_id=info:doi\/10.1021%2Facs.jafc.5b02105&rft_id=info:pmid\/26168136&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15_25-1\">25.1<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15_25-2\">25.2<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezDeterm15_25-3\">25.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mart\u00ednez-Dom\u00ednguez, G.; Romero-Gonz\u00e1lez, R.; Garrido Frenich, A. (2015). \"Determination of toxic substances, pesticides and mycotoxins, in ginkgo biloba nutraceutical products by liquid chromatography Orbitrap-mass spectrometry\". <i>Microchemical Journal<\/i> <b>118<\/b>: 124\u201330. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.microc.2014.09.002\" target=\"_blank\">10.1016\/j.microc.2014.09.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determination+of+toxic+substances%2C+pesticides+and+mycotoxins%2C+in+ginkgo+biloba+nutraceutical+products+by+liquid+chromatography+Orbitrap-mass+spectrometry&rft.jtitle=Microchemical+Journal&rft.aulast=Mart%C3%ADnez-Dom%C3%ADnguez%2C+G.%3B+Romero-Gonz%C3%A1lez%2C+R.%3B+Garrido+Frenich%2C+A.&rft.au=Mart%C3%ADnez-Dom%C3%ADnguez%2C+G.%3B+Romero-Gonz%C3%A1lez%2C+R.%3B+Garrido+Frenich%2C+A.&rft.date=2015&rft.volume=118&rft.pages=124%E2%80%9330&rft_id=info:doi\/10.1016%2Fj.microc.2014.09.002&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMulti16-26\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMulti16_26-0\">26.0<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMulti16_26-1\">26.1<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMulti16_26-2\">26.2<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMulti16_26-3\">26.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mart\u00ednez-Dom\u00ednguez, G.; Romero-Gonz\u00e1lez, R.; Garrido Frenich, A. (2016). \"Multi-class methodology to determine pesticides and mycotoxins in green tea and royal jelly supplements by liquid chromatography coupled to Orbitrap high resolution mass spectrometry\". <i>Food Chemistry<\/i> <b>197<\/b> (Pt. A): 907-15. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.foodchem.2015.11.070\" target=\"_blank\">10.1016\/j.foodchem.2015.11.070<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26617033\" target=\"_blank\">26617033<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multi-class+methodology+to+determine+pesticides+and+mycotoxins+in+green+tea+and+royal+jelly+supplements+by+liquid+chromatography+coupled+to+Orbitrap+high+resolution+mass+spectrometry&rft.jtitle=Food+Chemistry&rft.aulast=Mart%C3%ADnez-Dom%C3%ADnguez%2C+G.%3B+Romero-Gonz%C3%A1lez%2C+R.%3B+Garrido+Frenich%2C+A.&rft.au=Mart%C3%ADnez-Dom%C3%ADnguez%2C+G.%3B+Romero-Gonz%C3%A1lez%2C+R.%3B+Garrido+Frenich%2C+A.&rft.date=2016&rft.volume=197&rft.issue=Pt.+A&rft.pages=907-15&rft_id=info:doi\/10.1016%2Fj.foodchem.2015.11.070&rft_id=info:pmid\/26617033&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16_27-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16_27-1\">27.1<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16_27-2\">27.2<\/a><\/sup> <sup><a href=\"#cite_ref-Mart.C3.ADnez-Dom.C3.ADnguezMultiSoy16_27-3\">27.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mart\u00ednez-Dom\u00ednguez, G.; Romero-Gonz\u00e1lez, R.; Arrebola, F.J. et al. (2016). \"Multi-class determination of pesticides and mycotoxins in isoflavones supplements obtained from soy by liquid chromatography coupled to Orbitrap high resolution mass spectrometry\". <i>Food Control<\/i> <b>59<\/b>: 218\u201324. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.foodcont.2015.05.033\" target=\"_blank\">10.1016\/j.foodcont.2015.05.033<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multi-class+determination+of+pesticides+and+mycotoxins+in+isoflavones+supplements+obtained+from+soy+by+liquid+chromatography+coupled+to+Orbitrap+high+resolution+mass+spectrometry&rft.jtitle=Food+Control&rft.aulast=Mart%C3%ADnez-Dom%C3%ADnguez%2C+G.%3B+Romero-Gonz%C3%A1lez%2C+R.%3B+Arrebola%2C+F.J.+et+al.&rft.au=Mart%C3%ADnez-Dom%C3%ADnguez%2C+G.%3B+Romero-Gonz%C3%A1lez%2C+R.%3B+Arrebola%2C+F.J.+et+al.&rft.date=2016&rft.volume=59&rft.pages=218%E2%80%9324&rft_id=info:doi\/10.1016%2Fj.foodcont.2015.05.033&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SantosScreen09-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SantosScreen09_28-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-SantosScreen09_28-1\">28.1<\/a><\/sup> <sup><a href=\"#cite_ref-SantosScreen09_28-2\">28.2<\/a><\/sup> <sup><a href=\"#cite_ref-SantosScreen09_28-3\">28.3<\/a><\/sup> <sup><a href=\"#cite_ref-SantosScreen09_28-4\">28.4<\/a><\/sup> <sup><a href=\"#cite_ref-SantosScreen09_28-5\">28.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Santos, L.; Mar\u00edn, S.; Sanchis, V. et al. (2009). \"Screening of mycotoxin multicontamination in medicinal and aromatic herbs sampled in Spain\". <i>Journal of the Science of Food and Agriculture<\/i> <b>89<\/b> (10): 1802\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fjsfa.3647\" target=\"_blank\">10.1002\/jsfa.3647<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Screening+of+mycotoxin+multicontamination+in+medicinal+and+aromatic+herbs+sampled+in+Spain&rft.jtitle=Journal+of+the+Science+of+Food+and+Agriculture&rft.aulast=Santos%2C+L.%3B+Mar%C3%ADn%2C+S.%3B+Sanchis%2C+V.+et+al.&rft.au=Santos%2C+L.%3B+Mar%C3%ADn%2C+S.%3B+Sanchis%2C+V.+et+al.&rft.date=2009&rft.volume=89&rft.issue=10&rft.pages=1802%E2%80%937&rft_id=info:doi\/10.1002%2Fjsfa.3647&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhaoDeterm17-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhaoDeterm17_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhao, H.; Chen, X.; Shen, C. et al. (2017). \"Determination of 16 mycotoxins in vegetable oils using a QuEChERS method combined with high-performance liquid chromatography-tandem mass spectrometry\". <i>Food Additives and Contaminates Part A<\/i> <b>34<\/b> (2): 255\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F19440049.2016.1266096\" target=\"_blank\">10.1080\/19440049.2016.1266096<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27892850\" target=\"_blank\">27892850<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determination+of+16+mycotoxins+in+vegetable+oils+using+a+QuEChERS+method+combined+with+high-performance+liquid+chromatography-tandem+mass+spectrometry&rft.jtitle=Food+Additives+and+Contaminates+Part+A&rft.aulast=Zhao%2C+H.%3B+Chen%2C+X.%3B+Shen%2C+C.+et+al.&rft.au=Zhao%2C+H.%3B+Chen%2C+X.%3B+Shen%2C+C.+et+al.&rft.date=2017&rft.volume=34&rft.issue=2&rft.pages=255%E2%80%9364&rft_id=info:doi\/10.1080%2F19440049.2016.1266096&rft_id=info:pmid\/27892850&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Hidalgo-RuizARapid19-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Hidalgo-RuizARapid19_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hidalgo-Ruiz, J.L;. Romero-Gonz\u00e1lez, R.; Mart\u00ednez Vidal, J.L. et al. (2019). \"A rapid method for the determination of mycotoxins in edible vegetable oils by ultra-high performance liquid chromatography-tandem mass spectrometry\". <i>Food Chemistry<\/i> <b>288<\/b>: 22\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.foodchem.2019.03.003\" target=\"_blank\">10.1016\/j.foodchem.2019.03.003<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30902285\" target=\"_blank\">30902285<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+rapid+method+for+the+determination+of+mycotoxins+in+edible+vegetable+oils+by+ultra-high+performance+liquid+chromatography-tandem+mass+spectrometry&rft.jtitle=Food+Chemistry&rft.aulast=Hidalgo-Ruiz%2C+J.L%3B.+Romero-Gonz%C3%A1lez%2C+R.%3B+Mart%C3%ADnez+Vidal%2C+J.L.+et+al.&rft.au=Hidalgo-Ruiz%2C+J.L%3B.+Romero-Gonz%C3%A1lez%2C+R.%3B+Mart%C3%ADnez+Vidal%2C+J.L.+et+al.&rft.date=2019&rft.volume=288&rft.pages=22%E2%80%938&rft_id=info:doi\/10.1016%2Fj.foodchem.2019.03.003&rft_id=info:pmid\/30902285&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MyresiotisDeterm15-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MyresiotisDeterm15_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Myresiotis, C.K.; Testempasis, S;. Vryzas, Z. et al. (2015). \"Determination of mycotoxins in pomegranate fruits and juices using a QuEChERS-based method\". <i>Food Chemistry<\/i> <b>182<\/b>: 81\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.foodchem.2015.02.141\" target=\"_blank\">10.1016\/j.foodchem.2015.02.141<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25842312\" target=\"_blank\">25842312<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determination+of+mycotoxins+in+pomegranate+fruits+and+juices+using+a+QuEChERS-based+method&rft.jtitle=Food+Chemistry&rft.aulast=Myresiotis%2C+C.K.%3B+Testempasis%2C+S%3B.+Vryzas%2C+Z.+et+al.&rft.au=Myresiotis%2C+C.K.%3B+Testempasis%2C+S%3B.+Vryzas%2C+Z.+et+al.&rft.date=2015&rft.volume=182&rft.pages=81%E2%80%938&rft_id=info:doi\/10.1016%2Fj.foodchem.2015.02.141&rft_id=info:pmid\/25842312&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HanEval16-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HanEval16_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Han, L;. Matarrita, J.; Sapozhnikova, Y. et al. (2015). \"Evaluation of a recent product to remove lipids and other matrix co-extractives in the analysis of pesticide residues and environmental contaminants in foods\". <i>Journal of Chromatography A<\/i> <b>182<\/b>: 81\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.foodchem.2015.02.141\" target=\"_blank\">10.1016\/j.foodchem.2015.02.141<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25842312\" target=\"_blank\">25842312<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+of+a+recent+product+to+remove+lipids+and+other+matrix+co-extractives+in+the+analysis+of+pesticide+residues+and+environmental+contaminants+in+foods&rft.jtitle=Journal+of+Chromatography+A&rft.aulast=Han%2C+L%3B.+Matarrita%2C+J.%3B+Sapozhnikova%2C+Y.+et+al.&rft.au=Han%2C+L%3B.+Matarrita%2C+J.%3B+Sapozhnikova%2C+Y.+et+al.&rft.date=2015&rft.volume=182&rft.pages=81%E2%80%938&rft_id=info:doi\/10.1016%2Fj.foodchem.2015.02.141&rft_id=info:pmid\/25842312&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RajskiDeterm13-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RajskiDeterm13_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rajski, \u0141.; Lozano, A;. Ucl\u00e9s, A. et al. (2013). \"Determination of pesticide residues in high oil vegetal commodities by using various multi-residue methods and clean-ups followed by liquid chromatography tandem mass spectrometry\". <i>Journal of Chromatography A<\/i> <b>1304<\/b>: 109\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.chroma.2013.06.070\" target=\"_blank\">10.1016\/j.chroma.2013.06.070<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23871559\" target=\"_blank\">23871559<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determination+of+pesticide+residues+in+high+oil+vegetal+commodities+by+using+various+multi-residue+methods+and+clean-ups+followed+by+liquid+chromatography+tandem+mass+spectrometry&rft.jtitle=Journal+of+Chromatography+A&rft.aulast=Rajski%2C+%C5%81.%3B+Lozano%2C+A%3B.+Ucl%C3%A9s%2C+A.+et+al.&rft.au=Rajski%2C+%C5%81.%3B+Lozano%2C+A%3B.+Ucl%C3%A9s%2C+A.+et+al.&rft.date=2013&rft.volume=1304&rft.pages=109%E2%80%9320&rft_id=info:doi\/10.1016%2Fj.chroma.2013.06.070&rft_id=info:pmid\/23871559&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LozanoEval14-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LozanoEval14_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lozano, A.; Rajski, \u0141.; Ucl\u00e9s, S. et al. (2014). \"Evaluation of zirconium dioxide-based sorbents to decrease the matrix effect in avocado and almond multiresidue pesticide analysis followed by gas chromatography tandem mass spectrometry\". <i>Talanta<\/i> <b>118<\/b>: 68\u201383. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.talanta.2013.09.053\" target=\"_blank\">10.1016\/j.talanta.2013.09.053<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24274272\" target=\"_blank\">24274272<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+of+zirconium+dioxide-based+sorbents+to+decrease+the+matrix+effect+in+avocado+and+almond+multiresidue+pesticide+analysis+followed+by+gas+chromatography+tandem+mass+spectrometry&rft.jtitle=Talanta&rft.aulast=Lozano%2C+A.%3B+Rajski%2C+%C5%81.%3B+Ucl%C3%A9s%2C+S.+et+al.&rft.au=Lozano%2C+A.%3B+Rajski%2C+%C5%81.%3B+Ucl%C3%A9s%2C+S.+et+al.&rft.date=2014&rft.volume=118&rft.pages=68%E2%80%9383&rft_id=info:doi\/10.1016%2Fj.talanta.2013.09.053&rft_id=info:pmid\/24274272&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TuzimskiMethod19-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TuzimskiMethod19_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tuzimski, T.; Szubartowski, S. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6600471\" target=\"_blank\">\"Method Development for Selected Bisphenols Analysis in Sweetened Condensed Milk from a Can and Breast Milk Samples by HPLC-DAD and HPLC-QqQ-MS: Comparison of Sorbents (Z-SEP, Z-SEP Plus, PSA, C18, Chitin and EMR-Lipid) for Clean-Up of QuEChERS Extract\"<\/a>. <i>Molecules<\/i> <b>24<\/b> (11): E2093. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fmolecules24112093\" target=\"_blank\">10.3390\/molecules24112093<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6600471\/\" target=\"_blank\">PMC6600471<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31159388\" target=\"_blank\">31159388<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6600471\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6600471<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Method+Development+for+Selected+Bisphenols+Analysis+in+Sweetened+Condensed+Milk+from+a+Can+and+Breast+Milk+Samples+by+HPLC-DAD+and+HPLC-QqQ-MS%3A+Comparison+of+Sorbents+%28Z-SEP%2C+Z-SEP+Plus%2C+PSA%2C+C18%2C+Chitin+and+EMR-Lipid%29+for+Clean-Up+of+QuEChERS+Extract&rft.jtitle=Molecules&rft.aulast=Tuzimski%2C+T.%3B+Szubartowski%2C+S.&rft.au=Tuzimski%2C+T.%3B+Szubartowski%2C+S.&rft.date=2019&rft.volume=24&rft.issue=11&rft.pages=E2093&rft_id=info:doi\/10.3390%2Fmolecules24112093&rft_id=info:pmc\/PMC6600471&rft_id=info:pmid\/31159388&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6600471&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EURegulation657_02-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-EURegulation657_02_36-0\">36.0<\/a><\/sup> <sup><a href=\"#cite_ref-EURegulation657_02_36-1\">36.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32002D0657\" target=\"_blank\">\"2002\/657\/EC: Commission Decision of 12 August 2002 implementing Council Directive 96\/23\/EC concerning the performance of analytical methods and the interpretation of results\"<\/a>. <i>EUR-Lex<\/i>. European Union. 17 August 2002<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32002D0657\" target=\"_blank\">https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX%3A32002D0657<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=2002%2F657%2FEC%3A+Commission+Decision+of+12+August+2002+implementing+Council+Directive+96%2F23%2FEC+concerning+the+performance+of+analytical+methods+and+the+interpretation+of+results&rft.atitle=EUR-Lex&rft.date=17+August+2002&rft.pub=European+Union&rft_id=https%3A%2F%2Feur-lex.europa.eu%2Flegal-content%2FEN%2FALL%2F%3Furi%3DCELEX%253A32002D0657&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HanMultiOf12-37\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HanMultiOf12_37-0\">37.0<\/a><\/sup> <sup><a href=\"#cite_ref-HanMultiOf12_37-1\">37.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Han, Z.; Ren, Y.; Zhu, J. et al. (2012). \"Multianalysis of 35 mycotoxins in traditional Chinese medicines by ultra-high-performance liquid chromatography-tandem mass spectrometry coupled with accelerated solvent extraction\". <i>Journal of Agricultural and Food Chemistry<\/i> <b>60<\/b> (33): 8233-47. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fjf301928r\" target=\"_blank\">10.1021\/jf301928r<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22823451\" target=\"_blank\">22823451<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multianalysis+of+35+mycotoxins+in+traditional+Chinese+medicines+by+ultra-high-performance+liquid+chromatography-tandem+mass+spectrometry+coupled+with+accelerated+solvent+extraction&rft.jtitle=Journal+of+Agricultural+and+Food+Chemistry&rft.aulast=Han%2C+Z.%3B+Ren%2C+Y.%3B+Zhu%2C+J.+et+al.&rft.au=Han%2C+Z.%3B+Ren%2C+Y.%3B+Zhu%2C+J.+et+al.&rft.date=2012&rft.volume=60&rft.issue=33&rft.pages=8233-47&rft_id=info:doi\/10.1021%2Fjf301928r&rft_id=info:pmid\/22823451&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TournasOccurr12-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TournasOccurr12_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tournas, V.H.; Sapp, C.; Trucksess, M.W. (2012). \"Occurrence of aflatoxins in milk thistle herbal supplements\". <i>Food Additives and Contaminated Part A.<\/i> <b>29<\/b> (6): 994\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F19440049.2012.664788\" target=\"_blank\">10.1080\/19440049.2012.664788<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22439650\" target=\"_blank\">22439650<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Occurrence+of+aflatoxins+in+milk+thistle+herbal+supplements&rft.jtitle=Food+Additives+and+Contaminated+Part+A.&rft.aulast=Tournas%2C+V.H.%3B+Sapp%2C+C.%3B+Trucksess%2C+M.W.&rft.au=Tournas%2C+V.H.%3B+Sapp%2C+C.%3B+Trucksess%2C+M.W.&rft.date=2012&rft.volume=29&rft.issue=6&rft.pages=994%E2%80%939&rft_id=info:doi\/10.1080%2F19440049.2012.664788&rft_id=info:pmid\/22439650&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Arroyo-ManzanaresMulti13-39\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Arroyo-ManzanaresMulti13_39-0\">39.0<\/a><\/sup> <sup><a href=\"#cite_ref-Arroyo-ManzanaresMulti13_39-1\">39.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Arroyo-Manzanares, N.; Garc\u00eda-Campa\u00f1a, A.M.; G\u00e1miz-Gracia, L. (2013). \"Multiclass mycotoxin analysis in Silybum marianum by ultra high performance liquid chromatography-tandem mass spectrometry using a procedure based on QuEChERS and dispersive liquid-liquid microextraction\". <i>Journal of Chromatography A.<\/i> <b>1282<\/b>: 11\u201319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.chroma.2013.01.072\" target=\"_blank\">10.1016\/j.chroma.2013.01.072<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23415469\" target=\"_blank\">23415469<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multiclass+mycotoxin+analysis+in+Silybum+marianum+by+ultra+high+performance+liquid+chromatography-tandem+mass+spectrometry+using+a+procedure+based+on+QuEChERS+and+dispersive+liquid-liquid+microextraction&rft.jtitle=Journal+of+Chromatography+A.&rft.aulast=Arroyo-Manzanares%2C+N.%3B+Garc%C3%ADa-Campa%C3%B1a%2C+A.M.%3B+G%C3%A1miz-Gracia%2C+L.&rft.au=Arroyo-Manzanares%2C+N.%3B+Garc%C3%ADa-Campa%C3%B1a%2C+A.M.%3B+G%C3%A1miz-Gracia%2C+L.&rft.date=2013&rft.volume=1282&rft.pages=11%E2%80%9319&rft_id=info:doi\/10.1016%2Fj.chroma.2013.01.072&rft_id=info:pmid\/23415469&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HanARapid11-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HanARapid11_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Han, Z.; Ren, Y.; Zhou, H. et al. (2011). \"A rapid method for simultaneous determination of zearalenone, \u03b1-zearalenol, \u03b2-zearalenol, zearalanone, \u03b1-zearalanol and \u03b2-zearalanol in traditional Chinese medicines by ultra-high-performance liquid chromatography-tandem mass spectrometry\". <i>Journal of Chromatography B.<\/i> <b>879<\/b> (5\u20136): 411\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jchromb.2010.12.028\" target=\"_blank\">10.1016\/j.jchromb.2010.12.028<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21242113\" target=\"_blank\">21242113<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+rapid+method+for+simultaneous+determination+of+zearalenone%2C+%CE%B1-zearalenol%2C+%CE%B2-zearalenol%2C+zearalanone%2C+%CE%B1-zearalanol+and+%CE%B2-zearalanol+in+traditional+Chinese+medicines+by+ultra-high-performance+liquid+chromatography-tandem+mass+spectrometry&rft.jtitle=Journal+of+Chromatography+B.&rft.aulast=Han%2C+Z.%3B+Ren%2C+Y.%3B+Zhou%2C+H.+et+al.&rft.au=Han%2C+Z.%3B+Ren%2C+Y.%3B+Zhou%2C+H.+et+al.&rft.date=2011&rft.volume=879&rft.issue=5%E2%80%936&rft.pages=411%E2%80%9320&rft_id=info:doi\/10.1016%2Fj.jchromb.2010.12.028&rft_id=info:pmid\/21242113&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmithNatural16-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmithNatural16_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smith, M.C.; Madec, S.; Coton, E. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4848621\" target=\"_blank\">\"Natural Co-Occurrence of Mycotoxins in Foods and Feeds and Their in vitro Combined Toxicological Effects\"<\/a>. <i>Toxins<\/i> <b>8<\/b> (4): 94. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Ftoxins8040094\" target=\"_blank\">10.3390\/toxins8040094<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4848621\/\" target=\"_blank\">PMC4848621<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27023609\" target=\"_blank\">27023609<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4848621\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4848621<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Natural+Co-Occurrence+of+Mycotoxins+in+Foods+and+Feeds+and+Their+in+vitro+Combined+Toxicological+Effects&rft.jtitle=Toxins&rft.aulast=Smith%2C+M.C.%3B+Madec%2C+S.%3B+Coton%2C+E.+et+al.&rft.au=Smith%2C+M.C.%3B+Madec%2C+S.%3B+Coton%2C+E.+et+al.&rft.date=2016&rft.volume=8&rft.issue=4&rft.pages=94&rft_id=info:doi\/10.3390%2Ftoxins8040094&rft_id=info:pmc\/PMC4848621&rft_id=info:pmid\/27023609&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4848621&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PPDBPest-42\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PPDBPest_42-0\">42.0<\/a><\/sup> <sup><a href=\"#cite_ref-PPDBPest_42-1\">42.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/sitem.herts.ac.uk\/aeru\/ppdb\/en\/\" target=\"_blank\">\"PPDB: Pesticide Properties DataBase\"<\/a>. University of Hertfordshire. 03 February 2007<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/sitem.herts.ac.uk\/aeru\/ppdb\/en\/\" target=\"_blank\">http:\/\/sitem.herts.ac.uk\/aeru\/ppdb\/en\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=PPDB%3A+Pesticide+Properties+DataBase&rft.atitle=&rft.date=03+February+2007&rft.pub=University+of+Hertfordshire&rft_id=http%3A%2F%2Fsitem.herts.ac.uk%2Faeru%2Fppdb%2Fen%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EURegulation1107_09-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EURegulation1107_09_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=celex%3A32009R1107\" target=\"_blank\">\"Regulation (EC) No 1107\/2009 of the European Parliament and of the Council of 21 October 2009 concerning the placing of plant protection products on the market and repealing Council Directives 79\/117\/EEC and 91\/414\/EEC\"<\/a>. <i>EUR-Lex<\/i>. European Union. 24 November 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=celex%3A32009R1107\" target=\"_blank\">https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=celex%3A32009R1107<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Regulation+%28EC%29+No+1107%2F2009+of+the+European+Parliament+and+of+the+Council+of+21+October+2009+concerning+the+placing+of+plant+protection+products+on+the+market+and+repealing+Council+Directives+79%2F117%2FEEC+and+91%2F414%2FEEC&rft.atitle=EUR-Lex&rft.date=24+November+2009&rft.pub=European+Union&rft_id=https%3A%2F%2Feur-lex.europa.eu%2Flegal-content%2FEN%2FTXT%2F%3Furi%3Dcelex%253A32009R1107&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EURegulation247_04-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EURegulation247_04_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX%3A32004D0247\" target=\"_blank\">\"2004\/247\/EC: Commission Decision of 10 March 2004 concerning the non-inclusion of simazine in Annex I to Council Directive 91\/414\/EEC and the withdrawal of authorisations for plant protection products containing this active substance\"<\/a>. <i>EUR-Lex<\/i>. European Union. 16 March 2004<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX%3A32004D0247\" target=\"_blank\">https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX%3A32004D0247<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=2004%2F247%2FEC%3A+Commission+Decision+of+10+March+2004+concerning+the+non-inclusion+of+simazine+in+Annex+I+to+Council+Directive+91%2F414%2FEEC+and+the+withdrawal+of+authorisations+for+plant+protection+products+containing+this+active+substance&rft.atitle=EUR-Lex&rft.date=16+March+2004&rft.pub=European+Union&rft_id=https%3A%2F%2Feur-lex.europa.eu%2Flegal-content%2FEN%2FTXT%2F%3Furi%3DCELEX%253A32004D0247&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CastaldoTarget19-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CastaldoTarget19_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Castaldo, L.; Graziani, G.; Gaspari, A. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6723864\" target=\"_blank\">\"Target Analysis and Retrospective Screening of Multiple Mycotoxins in Pet Food Using UHPLC-Q-Orbitrap HRMS\"<\/a>. <i>Toxins<\/i> <b>11<\/b> (8): 411\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Ftoxins11080434\" target=\"_blank\">10.3390\/toxins11080434<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6723864\/\" target=\"_blank\">PMC6723864<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31344880\" target=\"_blank\">31344880<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6723864\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6723864<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Target+Analysis+and+Retrospective+Screening+of+Multiple+Mycotoxins+in+Pet+Food+Using+UHPLC-Q-Orbitrap+HRMS&rft.jtitle=Toxins&rft.aulast=Castaldo%2C+L.%3B+Graziani%2C+G.%3B+Gaspari%2C+A.+et+al.&rft.au=Castaldo%2C+L.%3B+Graziani%2C+G.%3B+Gaspari%2C+A.+et+al.&rft.date=2019&rft.volume=11&rft.issue=8&rft.pages=411%E2%80%9320&rft_id=info:doi\/10.3390%2Ftoxins11080434&rft_id=info:pmc\/PMC6723864&rft_id=info:pmid\/31344880&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6723864&rfr_id=info:sid\/en.wikipedia.org:Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added. The references are slightly out of order compared to the original, starting at reference 38, due to non-sequential ordering in the original Table 2.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204357\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.869 seconds\nReal time usage: 2.926 seconds\nPreprocessor visited node count: 35169\/1000000\nPreprocessor generated node count: 41417\/1000000\nPost\u2010expand include size: 316289\/2097152 bytes\nTemplate argument size: 108352\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 730.003 1 - -total\n 87.27% 637.066 1 - Template:Reflist\n 73.54% 536.831 45 - Template:Citation\/core\n 62.54% 456.561 35 - Template:Cite_journal\n 10.82% 78.968 8 - Template:Cite_web\n 8.86% 64.710 75 - Template:Citation\/identifier\n 5.30% 38.674 1 - Template:Infobox_journal_article\n 5.05% 36.840 1 - Template:Infobox\n 4.21% 30.721 54 - Template:Citation\/make_link\n 3.68% 26.828 2 - Template:Cite_book\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11426-0!*!0!!en!5!* and timestamp 20200707204355 and revision id 38125\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals\">https:\/\/www.limswiki.org\/index.php\/Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","3eda7d7fb752e96c7816c5992924922e_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2b\/Fig1_Narv%C3%A1ez_Toxins2020_12-2.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/07\/Tab1_Narv%C3%A1ez_Toxins2020_12-2.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/28\/Tab3_Narv%C3%A1ez_Toxins2020_12-2.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b7\/Fig2_Narv%C3%A1ez_Toxins2020_12-2.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/9a\/Tab4_Narv%C3%A1ez_Toxins2020_12-2.png"],"3eda7d7fb752e96c7816c5992924922e_timestamp":1594154635,"9d43e21cd08ee50665979d490d94d5ef_type":"article","9d43e21cd08ee50665979d490d94d5ef_title":"Project management in laboratory medicine (Lippi and Mattiuzi 2019)","9d43e21cd08ee50665979d490d94d5ef_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Project_management_in_laboratory_medicine","9d43e21cd08ee50665979d490d94d5ef_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Project management in laboratory medicine\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nProject management in laboratory medicineJournal\n \nJournal of Medical BiochemistryAuthor(s)\n \nLippi, Guiseppe; Mattiuzi, CamillaAuthor affiliation(s)\n \nUniversity of Verona, Provincial Agency for Social and Sanitary Services (Trento, Italy)Primary contact\n \nEmail: giuseppe dot lippi at univr dot itYear published\n \n2019Volume and issue\n \n38(4)Page(s)\n \n401\u20136DOI\n \n10.2478\/jomb-2019-0021ISSN\n \n1452-8266Distribution license\n \nCreative Commons Attribution-NonCommercial-NoDerivatives 4.0 InternationalWebsite\n \nhttps:\/\/content.sciendo.com\/contentpage\/Download\n \nhttps:\/\/content.sciendo.com\/downloadpdf\/journals\/jomb\/38\/4\/article-p401.xml (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Step 1 - Defining the environment \n4 Step 2 - Planning technical resources \n5 Step 3 - Staff availability and qualification \n6 Step 4 - Interplay with hospital administration \n7 Conclusions \n8 Acknowledgements \n\n8.1 Conflict of interest \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nThe role and responsibilities of laboratory managers have considerably evolved during the past decades. This revolution has been mostly driven by biological, technical, economic, and social factors, such as a deepened understanding of the pathophysiology of human diseases, technical innovations, renewed focus on patient safety, cost-containment strategies and patient empowerment. One of the leading consequences is an ongoing process of reorganization, consolidation, and automation of laboratory services, whose propitious realization strongly relies on establishing an efficient project management plan. In a practical perspective, the leading drivers of project management in laboratory medicine encompass various activities supporting a clear definition of the local environment, an accurate planning of technical resources, the acknowledgement of staff availability and qualification, along with the establishment of a positive and constructive interplay with hospital administrators. Therefore, the aim of this article is to provide a personal overview on the main drivers and outcomes of project management in laboratory medicine, which will expectedly contribute to construct a new consciousness and an innovative and multifaceted job description of laboratory professionals worldwide.\nKeywords: laboratory medicine, diagnostic testing, project management, automation\n\nIntroduction \nLaboratory medicine is conventionally defined as a science committed to generate clinical information through analysis of concentration, composition, and\/or structure of different analytes in different biological fluids.[1] To be thoughtfully capable of providing a valued contribution to clinical decision making, laboratory medicine services shall hence be developed and organized for maximizing productive efficiency and optimizing clinical efficacy. Unlike many years ago, when healthcare services were not so strongly plagued by shortage of funding and could benefit from ample economic resources, the current scenario is now overwhelmed by an unprecedented worldwide economic crisis[2], which has also obligated laboratory managers to increase volume and complexity of testing, contextually preserving quality and cutting down costs. This altered scenario has inevitably forced laboratory managers and laboratory professionals to become familiar with many different tools borrowed from other professions, such as leadership skills[3], budgeting activities[4] and, last but not least, project management.\nAccording to a common inception, project management can be defined as the practice of initiating, planning, executing, monitoring and closing a specific work, aimed at achieving specific goals at a specified time. Project management is hence conventionally dictated by six main paradigms: efficiency, efficacy, quality, safety, sustainability, and satisfaction. The practical translation of these essential factors in the field of laboratory medicine is summarized in Table 1. Briefly, efficiency implies achieving maximum laboratory productivity with minimum wasted effort or expense, while efficacy is mainly directed towards improving diagnoses and clinical outcomes. Quality encompasses reaching the highest possible degree of reliability and safety of laboratory data, safety develops through limiting the risk of injury or damage to patients and staff, and sustainability requires avoiding depletion of human and economic resources. Finally, satisfaction is achieved by fulfilling wishes, expectations, or needs of both laboratory staff and its stakeholders (i.e., patients and doctors). \n\n\n\n\n\n\n\nTable 1. The six paradigms of project management in laboratory medicine\n\n\n\nParadigm\n\nDescription\n\n\nEfficiency\n\nTo achieve maximum laboratory productivity with minimum wasted effort or expense\n\n\nEfficacy\n\nTo achieve better diagnoses and improved clinical outcomes\n\n\nQuality\n\nTo develop the highest possible degree of reliability and safety in test results\n\n\nSafety\n\nTo limit the risk of injury or damage to patients and laboratory staff\n\n\nSustainability\n\nTo avoid depleting human and economic resources\n\n\nSatisfaction\n\nTo fulfill both laboratory staff and stakeholders\u2019 (i.e., patients\u2019, doctors\u2019) wishes, expectations, and needs\n\n\n\nFrom a practical perspective, the main drivers of project management in laboratory medicine encompass some fundamental but not essentially sequential steps, which entail a clear definition of the environment, an accurate planning of technical resources, the acknowledgement of staff availability and qualification, and the establishment of a positive and constructive interplay with hospital administrators. These steps are now further detailed.\n\nStep 1 - Defining the environment \nAs laboratory medicine continues to evolve from the performance of many manual activities towards the automation of several steps throughout the total testing process[5], so-called open-plan layouts are becoming commonplace to efficiently respond to the emerging issue of connecting many laboratory analyzers within the same system and developing an efficient workflow, from arrival of samples in the laboratory to their final discharge or storage once testing has been completed.[6] Space availability and organization shall hence be regarded as major limiting steps when projecting the final layout, since the preexisting environment may not be suited to accommodate multiple laboratory analyzers and their connecting systems within the available space. Although the possibility to start from zero (i.e., constructing a new purpose-built structure) is indeed the most desirable and advisable scenario, this rarely happens since the reorganization of most laboratory services goes through cosmetic rearrangements or modernization of preexisting buildings.[7] This would actually force laboratory managers to find a reasonable way to fit the elephant (i.e., automated laboratory instrumentation) into the room (i.e., preexisting environment). Indeed, many different solutions have been made available after the development of flexible laboratory automation, spanning from narrow automation of diagnostic areas (i.e., automation of clinical chemistry and\/or immunochemistry testing), up to complete automation of the largest part of laboratory diagnostics (i.e., total laboratory automation; TLA). The choice between the many available solutions of laboratory automation is dependent on the available space for connecting multiple instrumentation and the residual (i.e., vital) space necessary for allowing the laboratory staff to work on the instrumentation and contextually perform maintenance or repairing, when these activities will be needed.\nWhatever solution can be finally implemented, laboratory managers must be aware of the risk of the so-called \"point of no return,\" which is defined by the impossibility of easily and inexpensively reorganizing an inefficient laboratory layout once this has been definitely developed. In the unfortunate case of the final project partially or totally proving inefficient and nonfunctional, changing the layout could lead to catastrophic economic consequence, or can even be unfeasible.\n\nStep 2 - Planning technical resources \nIn the complex effort of planning the technical resources needed for achieving a given target (i.e., constructing a new laboratory layout), developing and documenting the project vision, mission, goals, and deliverables are essential prerequisites. More often than not, these activities are overlooked or completely ignored, whilst the vision and mission of the clinical laboratory should be aligned with those of the complex organization where the laboratory operates. Not ably, laboratory services are now frequently organized in networks, with the reference center in the middle (i.e., the so-called \"hub\" facility) and many decentralized laboratories in periphery (i.e., the so-called \"spokes\"), interconnected with an efficient system of sample deliverance and a versatile laboratory information system (LIS).[8][9] This actual organization requires developing the clear-cut concepts of clinical-laboratory liaison and diagnostic stewardship, according to which the laboratory shall be engaged in reorganizing its structure (instrumentation, tests, staff) for providing an effective technical and advisory support to the local clinical needs within the network, whilst clinicians shall fairly cooperate with the laboratory staff for identifying the most efficient and effective solutions according to location and resources availability.[10] An optimal balance should hence be always identified between case-mix (i.e., clinical complexity) of the healthcare facility where the laboratory is set and the locally available panel of diagnostic tests.\nAs an example, a regional reference center for management of severe bleeding disorders shall be equipped with a clinical laboratory capable to perform second-line and even third-line hemostasis tests, whilst a peripheral hospital within the same network would only need a basic hemostasis laboratory, since patients with severe bleeding disorder, either congenital or acquired, would be generally admitted and managed elsewhere.[11] In the case that a patient with a bleeding syndrome is brought to a peripheral facility, the local \"spoke\" laboratory can then support clinicians with a panel of first-line (screening) hemostasis tests (e.g., prothrombin time, activated partial thromboplastin time, fibrinogen, platelet count, screening of platelet function). In most cases these tests\u2014along with the clinical history, signs, and symptoms\u2014will be sufficient to guide clinical decision making and the decision as to whether the patient may need to be referred to the reference center (where the \"hub\" laboratory is available) for being further investigated and eventually managed, or can else be locally treated or safely discharged. This paradigmatic example can then be translated to the vast majority of laboratory medicine areas (e.g., hematology, immunochemistry, microbiology), by defining a clear hierarchy of tests that should be available in the different laboratories operating within a network. Regardless of personal inclinations and interests, \"spoke\" laboratories will generally need to be equipped with basic (first-line) laboratory tests, whereas \"hub\" laboratories will require more complex, time-consuming, and expensive (second- and third-line) analyses. As previously discussed, decisions on the final organization of laboratory diagnostics within a network of laboratory services must be taken in accordance with clinicians and hospital administrators, thus fulfilling clinical needs[12], principles of cost-effectiveness[13], and pre-analytical requirements.[14] Dissipating both human and economic resources for performing obsolete, redundant, clinically questionable, or potentially unreliable analyses would not be beneficial for the healthcare system as a whole.\nAn accurate plan of technical resources will hence encompass a thorough analysis of the local situation, which will then influence the design of laboratory layout, preferably driven by validated tools such as Lean management systems, which contextually enable to maximize efficiency and create a culture of continuous improvement.[15] The leading factors associated with this process are volume and complexity (i.e., capacity), equipment and utility lists, staffing model, work schedule, regulatory considerations, safety ergonomic requirements, and, given space availability, research and education considerations (i.e., in academic centers).\n\nStep 3 - Staff availability and qualification \nWhether this third step should follow or anticipate the planning of technical resources remains debated. This is mostly due to a recent metamorphosis that has occurred in staff availability. Personnel requirements have basically evolved from a demand conditioned by workflow, complexity and environment to a new scenario where shortage of public healthcare funding has contributed to make environment and staff availability (and qualification) the leading drivers of workflow and complexity. In 2014, a statistics of the World Health Organization (WHO) has alarmingly highlighted that the global shortage of doctors, nurses, midwives, and other healthcare professionals had already reached a 4.3 million deficit around the world.[16] The situation has worsened in recent years, so that the predicted worldwide shortage of health care workers will probably exceed 15 million by the year 2030.[17] Laboratory medicine makes no exception to this rule, since inefficient turnover has involved almost each category of laboratory professionals, especially during the past decade.[18] It is hence rather understandable that laboratory managers must place staff availability among the top list of drivers of project management. Rome wasn\u2019t built in a day, though it would have never been built without a huge and skilled Roman workforce. \nMutatis mutandis, volume, and complexity of laboratory testing must accurately be commensurate to the local availability of staff and to specific personnel education and qualifications. Importantly, when the available human resources do not meet predefined requirements of workforce and skills, additional strategies are required. These additional strategies must address further elimination of manual activities, automation of additional parts of the total testing process, and expanded consolidation of many different diagnostic areas, including the worst case scenario of reducing volume and complexity of diagnostic testing, or even outsourcing tests to private facilities.[13][19] Among the possible solutions, this last option has recently gained significant momentum and has become especially appealing for some healthcare administrators, who are seeking to save money by cutting down laboratory funding and externalizing large volumes of tests. Whether or not this strategy is cost-effective remains largely disputed, although recently published evidence attests that outsourcing laboratory services decreases sample quality, increases turnaround time, and enhances the overall risk of diagnostic errors.[20] Sizeable privatization of diagnostic testing is neither a clear-cut solution to the problem, since no reliable evidence has been provided that this will generate improved services and overall cost savings. Moreover, many doubts have been raised around the fact that private contractors do not need to openly disclose how public health money is spent, allocated, or collected. The meteoric ascent of Theranos in the firmament of laboratory medicine, followed by its rapid downfall, has taught us to be cautious in moving towards certain types of deregulated diagnostic testing.[21]\nNotably, critical issues in staff regulations (e.g., time on turn, recovery) must be clearly identified, and staff necessity should then be defined accordingly. A final consideration concerning personnel is that laboratory directors cannot usually select the staff. As such, they are rather constrained to develop leadership skills which will enable them to manage the existing personnel, thus placing the right person in the right place for doing the right activity at the right time. This obviously entails accurately knowing the potential personnel (i.e., weakness and strengths), trying to fulfill personal inclinations (whenever possible), and, especially, not blaming people when something goes wrong, since errors are frequently caused by a system failure rather than by individual mistakes.\n\nStep 4 - Interplay with hospital administration \nAs already anticipated in some previous parts of this article, laboratory professionals are increasingly involved in administrative duties, mostly encompassing test menu optimization, delivering training or education, and administering budgets.[22] It is increasingly essential that laboratory directors and managers have a profound understanding of the budget of their laboratory and use that information for developing appropriate strategies for responding to a clinical demand, learning to manage budgets on the basis of a cost model, and presenting enough details to meet the needs of financial managers. These many aspects have become virtually unavoidable because laboratory diagnostics is now assimilated to many other economic industries by policymakers and administrators and is hence subjected to scale economy and evaluated accordingly. To put it simply, laboratory managers should aim to establish a favorable and constructive interplay with hospital administrators. Laboratory managers will also need to learn the language of hospital administrators and policymakers, since it is highly unlikely that these two categories will be ever strongly committed to speak a clinical language.[23]\nRegardless of any local organization, however, it is now undeniable that the future of laboratory medicine will be mostly driven by national healthcare policies, which are typically defined by a number of paradigms such as the amount of public funding for in vitro diagnostic testing, health insurance strategies, and test reimbursement policies. On a local basis, it will become increasingly essential to define medium- and long-term trajectories with hospital administrators, especially in terms of reorganization of the healthcare network (which will then influence number and size of laboratory services), number of hospital beds and outpatient flow (which will then influence test volume), and case-mix evolution (which will then influence test menu). Knowing this information in advance is necessary for developing an efficient and effective project management plan in laboratory medicine.\n\nConclusions \nSeveral lines of evidence now attest that role and responsibilities of laboratory managers have considerably evolved over the last few decades.[24] These paradigm shifts have been mostly driven by clinical, technical, economic, and social factors, mainly encompassing deepened understanding of the pathophysiology of human diseases, technological innovations, renewed focus on patient safety, cost-containment strategies, and patient empowerment. \nThe most obvious consequence has been the development of an ongoing process of reorganization, consolidation, and automation of laboratory services. The effective realization of those services requires defining an efficient project management plan, as well as constructing a new consciousness and an innovative and multifaceted job description of laboratory professionals worldwide. Notably, some other important drivers and outcome measures shall be considered when restructuring or redesigning the layout of a laboratory service, as briefly summarized in Table 2. These essentially include the identification and management of potential political or ideological resistances to the changes; the need to share the strategic plan with laboratory staff, local authorities, syndicates, and stakeholders (i.e., clinicians and patients); the definition of reliable performance indicators (both qualitative and quantitative), which will help assessing as to whether the new project is efficient and effective; and continuous monitoring of staff and stakeholder satisfaction. \n\n\n\n\n\n\n\nTable 2. Leading drivers of project management in laboratory medicine\n\n\n\nStep 1 - Defining the environment\n\n\n\na. Define space availability and organization.\r\n\nb. Identify the most suitable local model of laboratory automation.\n\n\n\nStep 2 \u2013 Planning technical resources\n\n\n\na. Define the geographical context (\"hub\" or \"spoke\" laboratory).\r\n\nb. Recognize local clinical educational and research needs.\r\n\nc. Identify predicted volume and complexity of tests.\r\n\nd. Develop clinical-laboratory liaison and diagnostic stewardship.\r\n\ne. Take advantage of using a Lean management system.\n\n\n\nStep 3 \u2013 Staff availability and qualification\n\n\n\na. Define the number of available persons.\r\n\nb. Acknowledge technical and clinical qualification of the staff.\r\n\nc. Adapt volume and complexity of testing to available staff.\r\n\nd. Identify critical issues in staff regulations (e.g., time on turn, recovery).\r\n\ne. Place the right person in the right place for doing the right activity at the right time.\r\n\nf. Safeguard personnel and patient safety.\n\n\n\nStep 4 \u2013 Interplay with hospital administration\n\n\n\na. Seek out a constructive dialogue with hospital administrators.\r\n\nb. Acknowledge the local political context.\r\n\nc. Be aware of local healthcare plans (e.g., reorganization of healthcare network, number of beds, evolution of case-mix).\r\n\nd. Become familiar with administrative duties and budgeting processes.\n\n\n\nStep 5 \u2013 Additional drivers\n\n\n\na. Identify political or ideological resistances to the changes.\r\n\nb. Share the strategic plan with laboratory staff, local authorities, syndicates, and stakeholders (i.e., clinicians and patients).\r\n\nc. Define reliable performance indicators (qualitative and quantitative).\r\n\nd. Continuously monitor staff and stakeholders satisfaction.\r\n\ne. Identify an alternative solution (i.e., a backup plan).\r\n\nf. Publicize results (when successful).\n\n\n\n\nFinally, additional consideration must be made. For example, common experience teaches that the delineation of an alternative solution\u2014should the primary solution not come to fruition\u2014may certainly be helpful to overcome possible technical failures of a new project. Whenever possible, the switch from the old to the new laboratory layout\u2014especially when entailing the use of novel instrumentation\u2014should not be irreversible. Additionally, the two solutions should be allowed to run in parallel for a certain period of time, at least until most of the possible problems have been identified and solved. Last but not least, provided that the final project proves successful, results must be publicized so that others may take profit from the local translation of favorable outcomes.\n\nAcknowledgements \nThis article is a detailed summary of an oral presentation delivered during the 15th Belgrade Symposium for Balkan Region, Belgrade (Serbia), 11\u201312 April, 2019.\n\nConflict of interest \nThe authors state that they have no conflicts of interest regarding the publication of this article.\n\nReferences \n\n\n\u2191 Lippi, G. (2019). \"The Irreplaceable Value of Laboratory Diagnostics: Four Recent Tests that have Revolutionized Clinical Practice\". EJIFCC 30 (1): 7\u201313. PMC PMC6416815. PMID 30881270. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6416815 .   \n\n\u2191 Lippi, G. (2018). \"Weighting healthcare efficiency against available resources: value is the goal\". Diagnosis 5 (2): 39\u201340. doi:10.1515\/dx-2018-0031. PMID 29858902.   \n\n\u2191 Majki\u0107-Singh, N. (2017). \"Laboratory Medicine Management: Leadership Skills for Effective Laboratory\". Journal of Medical Biochemistry 36 (3): 207\u201310. doi:10.1515\/jomb-2017-0034. PMC PMC6287220. PMID 30564056. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287220 .   \n\n\u2191 Price, C.P.; Wolstenholme, J.; McGinley, P. et al. (2018). \"Translational health economics: The key to accountable adoption of in vitro diagnostic technologies\". Health Services Management Research 31 (1): 43\u201350. doi:10.1177\/0951484817736727. PMID 29084478.   \n\n\u2191 Hawker, C.D. (2017). \"Nonanalytic Laboratory Automation: A Quarter Century of Progress\". Clinical Chemistry 63 (6): 1074-1082. doi:10.1373\/clinchem.2017.272047. PMID 28396562.   \n\n\u2191 Genzen, J.R.; Burnham, C.D.; Felder, R.A. et al. (2018). \"Challenges and Opportunities in Implementing Total Laboratory Automation\". Clinical Chemistry 64 (2): 259\u201364. doi:10.1373\/clinchem.2017.274068. PMID 28971983.   \n\n\u2191 Lippi, G.; Da Rin, G. (2019). \"Advantages and limitations of total laboratory automation: A personal overview\". Clinical Chemistry and Laboratory Medicine 57 (6): 802\u201311. doi:10.1515\/cclm-2018-1323. PMID 30710480.   \n\n\u2191 Lippi, G.; Simundic, A.M. (2012). \"Laboratory networking and sample quality: A still relevant issue for patient safety\". Clinical Chemistry and Laboratory Medicine 50 (10): 1703\u20135. doi:10.1515\/cclm-2012-0245. PMID 23089698.   \n\n\u2191 Luki\u0107, V. (2017). \"Laboratory Information System - Where are we Today?\". Journal of Medical Biochemistry 36 (3): 220-224. doi:10.1515\/jomb-2017-0021. PMC PMC6287214. PMID 30564059. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287214 .   \n\n\u2191 Plebani, M.; Laposata, M;. Lippi, G. (2019). \"Driving the route of laboratory medicine: A manifesto for the future\". Internal and Emergency Medicine 14 (3): 337\u201340. doi:10.1007\/s11739-019-02053-z. PMID 30783946.   \n\n\u2191 Lippi, G., Favaloro, E.J. (2018). \"Laboratory hemostasis: From biology to the bench\". Clinical Chemistry and Laboratory Medicine 56 (7): 1035\u201345. doi:10.1515\/cclm-2017-1205. PMID 29455188.   \n\n\u2191 J\u00f8rgensen, P.E. (2017). \"Leadership and Management in Clinical Biochemistry\". Journal of Medical Biochemistry 36 (3): 216\u201319. doi:10.1515\/jomb-2017-0023. PMC PMC6287210. PMID 30564058. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287210 .   \n\n\u2191 13.0 13.1 Lippi, G.; Plebani, M. (2018). \"Cost, profitability and value of laboratory diagnostics: In God we trust, all others bring data\". Journal of Laboratory Medicine 43 (1): 1\u20133. doi:10.1515\/labmed-2018-0321.   \n\n\u2191 Lippi, G., Cadamuro, J. (2017). \"Novel Opportunities for Improving the Quality of Preanalytical Phase. A Glimpse to the Future?\". Journal of Medical Biochemistry 36 (4): 293-300. doi:10.1515\/jomb-2017-0029. PMC PMC6294089. PMID 30581325. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6294089 .   \n\n\u2191 Knowles, S.; Barnes, I. (2013). \"Lean laboratories: Laboratory medicine needs to learn from other industries how to deliver more for less\". Journal of Clinical Pathology 66 (8): 635\u20137. doi:10.1136\/jclinpath-2013-201624. PMID 23681948.   \n\n\u2191 Aluttis, C.; Bishaw, T.; Frank, M.W. (2014). \"The workforce for health in a globalized context--global shortages and international migration\". Global Health Action 7: 23611. doi:10.3402\/gha.v7.23611. PMC PMC3926986. PMID 24560265. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3926986 .   \n\n\u2191 Liu, J.X.; Goryakin, Y.; Maeda, A. et al. (2017). \"Global Health Workforce Labor Market Projections for 2030\". Human Resources for Health 15 (1): 11. doi:10.1186\/s12960-017-0187-2. PMC PMC5291995. PMID 28159017. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5291995 .   \n\n\u2191 Cortelyou-Ward, K.; Ramirez, B.; Rotarius, T. (2011). \"The laboratory workforce shortage: A managerial perspective\". The Health Care Manager 30 (2): 148\u201355. doi:10.1097\/HCM.0b013e318216f5df. PMID 21537137.   \n\n\u2191 Lippin, G.; Plebani, M. (2017). \"The add value of laboratory diagnostics: The many reasons why decision-makers should actually care\". Journal of Laboratory and Precision Medicine 2 (12): 100. doi:10.21037\/jlpm.2017.12.07.   \n\n\u2191 Chasin, B.S.; Elliott, S.P.; Klotz, S.A. (2007). \"Medical errors arising from outsourcing laboratory and radiology services\". American Journal of Medicine 120 (9): e9-11. doi:10.1016\/j.amjmed.2006.07.024. PMID 17765055.   \n\n\u2191 Lackner, K.J.; Gillery, P.; Lippi, G. et al. (2016). \"The Theranos phenomenon, scientific transparency and freedom of speech\". Clinical Chemistry and Laboratory Medicine 54 (9): 1403-5. doi:10.1515\/cclm-2016-0520. PMID 27442369.   \n\n\u2191 Plebani, M.; Laposata, M.; Lippi, G. (2019). \"A manifesto for the future of laboratory medicine professionals\". Clinica Chimica Acta 489: 49\u201352. doi:10.1016\/j.cca.2018.11.021. PMID 30445032.   \n\n\u2191 Horvath, A.R. (2013). \"From evidence to best practice in laboratory medicine\". Clinical Biochemist Reviews 34 (2): 47\u201360. PMC PMC3799219. PMID 24151341. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3799219 .   \n\n\u2191 Jassam, N.; Lake, J.; Dabrowska, M. et al. (2018). \"The European Federation of Clinical Chemistry and Laboratory Medicine syllabus for postgraduate education and training for Specialists in Laboratory Medicine: Version 5 - 2018\". Clinical Chemistry and Laboratory Medicine 56 (11): 1846\u201363. doi:10.1515\/cclm-2018-0344. PMID 29870392.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference. Otherwise, in accordance with the NoDerivatives portion of the original license, nothing else has been changed.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Project_management_in_laboratory_medicine\">https:\/\/www.limswiki.org\/index.php\/Journal:Project_management_in_laboratory_medicine<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on laboratory medicine\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 3 March 2020, at 00:26.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 322 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","9d43e21cd08ee50665979d490d94d5ef_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Project_management_in_laboratory_medicine skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Project management in laboratory medicine<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>The role and responsibilities of <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> managers have considerably evolved during the past decades. This revolution has been mostly driven by biological, technical, economic, and social factors, such as a deepened understanding of the pathophysiology of human diseases, technical innovations, renewed focus on patient safety, cost-containment strategies and patient empowerment. One of the leading consequences is an ongoing process of reorganization, consolidation, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_automation\" title=\"Laboratory automation\" class=\"wiki-link\" data-key=\"0061880849aeaca05f8aa27ae171f331\">automation<\/a> of laboratory services, whose propitious realization strongly relies on establishing an efficient project management plan. In a practical perspective, the leading drivers of project management in laboratory medicine encompass various activities supporting a clear definition of the local environment, an accurate planning of technical resources, the acknowledgement of staff availability and qualification, along with the establishment of a positive and constructive interplay with <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> administrators. Therefore, the aim of this article is to provide a personal overview on the main drivers and outcomes of project management in laboratory medicine, which will expectedly contribute to construct a new consciousness and an innovative and multifaceted job description of laboratory professionals worldwide.\n<\/p><p><b>Keywords<\/b>: laboratory medicine, diagnostic testing, project management, automation\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Laboratory medicine is conventionally defined as a science committed to generate clinical <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> through analysis of concentration, composition, and\/or structure of different analytes in different biological fluids.<sup id=\"rdp-ebb-cite_ref-LippiTheIrr19_1-0\" class=\"reference\"><a href=\"#cite_note-LippiTheIrr19-1\">[1]<\/a><\/sup> To be thoughtfully capable of providing a valued contribution to clinical decision making, laboratory medicine services shall hence be developed and organized for maximizing productive efficiency and optimizing clinical efficacy. Unlike many years ago, when healthcare services were not so strongly plagued by shortage of funding and could benefit from ample economic resources, the current scenario is now overwhelmed by an unprecedented worldwide economic crisis<sup id=\"rdp-ebb-cite_ref-LippiWeigh18_2-0\" class=\"reference\"><a href=\"#cite_note-LippiWeigh18-2\">[2]<\/a><\/sup>, which has also obligated <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> managers to increase volume and complexity of <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_test\" title=\"Medical test\" class=\"wiki-link\" data-key=\"0e5a509a721423b5cae83265bbb80c85\">testing<\/a>, contextually preserving <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_(business)\" title=\"Quality (business)\" class=\"wiki-link\" data-key=\"c4ac43430d1c3a3a15d1255257aaea37\">quality<\/a> and cutting down costs. This altered scenario has inevitably forced laboratory managers and laboratory professionals to become familiar with many different tools borrowed from other professions, such as leadership skills<sup id=\"rdp-ebb-cite_ref-Majki.C4.87-SinghLab17_3-0\" class=\"reference\"><a href=\"#cite_note-Majki.C4.87-SinghLab17-3\">[3]<\/a><\/sup>, budgeting activities<sup id=\"rdp-ebb-cite_ref-PriceTrans18_4-0\" class=\"reference\"><a href=\"#cite_note-PriceTrans18-4\">[4]<\/a><\/sup> and, last but not least, project management.\n<\/p><p>According to a common inception, project management can be defined as the practice of initiating, planning, executing, monitoring and closing a specific work, aimed at achieving specific goals at a specified time. Project management is hence conventionally dictated by six main paradigms: efficiency, efficacy, quality, safety, sustainability, and satisfaction. The practical translation of these essential factors in the field of laboratory medicine is summarized in Table 1. Briefly, efficiency implies achieving maximum laboratory productivity with minimum wasted effort or expense, while efficacy is mainly directed towards improving diagnoses and clinical outcomes. Quality encompasses reaching the highest possible degree of reliability and safety of laboratory data, safety develops through limiting the risk of injury or damage to patients and staff, and sustainability requires avoiding depletion of human and economic resources. Finally, satisfaction is achieved by fulfilling wishes, expectations, or needs of both laboratory staff and its stakeholders (i.e., patients and doctors). \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> The six paradigms of project management in laboratory medicine\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Paradigm\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Description\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Efficiency\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">To achieve maximum laboratory productivity with minimum wasted effort or expense\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Efficacy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">To achieve better diagnoses and improved clinical outcomes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Quality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">To develop the highest possible degree of reliability and safety in test results\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Safety\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">To limit the risk of injury or damage to patients and laboratory staff\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sustainability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">To avoid depleting human and economic resources\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Satisfaction\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">To fulfill both laboratory staff and stakeholders\u2019 (i.e., patients\u2019, doctors\u2019) wishes, expectations, and needs\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>From a practical perspective, the main drivers of project management in laboratory medicine encompass some fundamental but not essentially sequential steps, which entail a clear definition of the environment, an accurate planning of technical resources, the acknowledgement of staff availability and qualification, and the establishment of a positive and constructive interplay with <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> administrators. These steps are now further detailed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Step_1_-_Defining_the_environment\">Step 1 - Defining the environment<\/span><\/h2>\n<p>As laboratory medicine continues to evolve from the performance of many manual activities towards the <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_automation\" title=\"Laboratory automation\" class=\"wiki-link\" data-key=\"0061880849aeaca05f8aa27ae171f331\">automation<\/a> of several steps throughout the total testing process<sup id=\"rdp-ebb-cite_ref-HawkerNon17_5-0\" class=\"reference\"><a href=\"#cite_note-HawkerNon17-5\">[5]<\/a><\/sup>, so-called open-plan layouts are becoming commonplace to efficiently respond to the emerging issue of connecting many laboratory analyzers within the same system and developing an efficient <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a>, from arrival of <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">samples<\/a> in the laboratory to their final discharge or storage once testing has been completed.<sup id=\"rdp-ebb-cite_ref-GenzenChall18_6-0\" class=\"reference\"><a href=\"#cite_note-GenzenChall18-6\">[6]<\/a><\/sup> Space availability and organization shall hence be regarded as major limiting steps when projecting the final layout, since the preexisting environment may not be suited to accommodate multiple laboratory analyzers and their <a href=\"https:\/\/www.limswiki.org\/index.php\/Interface_(computing)\" title=\"Interface (computing)\" class=\"wiki-link\" data-key=\"2ff176595c6c4a8f06d27f35fd27aeec\">connecting systems<\/a> within the available space. Although the possibility to start from zero (i.e., constructing a new purpose-built structure) is indeed the most desirable and advisable scenario, this rarely happens since the reorganization of most laboratory services goes through cosmetic rearrangements or modernization of preexisting buildings.<sup id=\"rdp-ebb-cite_ref-LippiAdvant19_7-0\" class=\"reference\"><a href=\"#cite_note-LippiAdvant19-7\">[7]<\/a><\/sup> This would actually force laboratory managers to find a reasonable way to fit the elephant (i.e., automated laboratory instrumentation) into the room (i.e., preexisting environment). Indeed, many different solutions have been made available after the development of flexible laboratory automation, spanning from narrow automation of <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_diagnosis\" title=\"Medical diagnosis\" class=\"wiki-link\" data-key=\"6fd078bb38b5c9089d7271b4ba20fe7c\">diagnostic<\/a> areas (i.e., automation of <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_chemistry\" title=\"Clinical chemistry\" class=\"wiki-link\" data-key=\"184d3433dd1f9dba149f42bc82234b8d\">clinical chemistry<\/a> and\/or immunochemistry testing), up to complete automation of the largest part of laboratory diagnostics (i.e., total laboratory automation; TLA). The choice between the many available solutions of laboratory automation is dependent on the available space for connecting multiple instrumentation and the residual (i.e., vital) space necessary for allowing the laboratory staff to work on the instrumentation and contextually perform maintenance or repairing, when these activities will be needed.\n<\/p><p>Whatever solution can be finally implemented, laboratory managers must be aware of the risk of the so-called \"point of no return,\" which is defined by the impossibility of easily and inexpensively reorganizing an inefficient laboratory layout once this has been definitely developed. In the unfortunate case of the final project partially or totally proving inefficient and nonfunctional, changing the layout could lead to catastrophic economic consequence, or can even be unfeasible.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Step_2_-_Planning_technical_resources\">Step 2 - Planning technical resources<\/span><\/h2>\n<p>In the complex effort of planning the technical resources needed for achieving a given target (i.e., constructing a new laboratory layout), developing and documenting the project vision, mission, goals, and deliverables are essential prerequisites. More often than not, these activities are overlooked or completely ignored, whilst the vision and mission of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_laboratory\" title=\"Clinical laboratory\" class=\"wiki-link\" data-key=\"307bcdf1bdbcd1bb167cee435b7a5463\">clinical laboratory<\/a> should be aligned with those of the complex organization where the laboratory operates. Not ably, laboratory services are now frequently organized in networks, with the reference center in the middle (i.e., the so-called \"hub\" facility) and many decentralized laboratories in periphery (i.e., the so-called \"spokes\"), interconnected with an efficient system of sample deliverance and a versatile <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS).<sup id=\"rdp-ebb-cite_ref-LippiLab12_8-0\" class=\"reference\"><a href=\"#cite_note-LippiLab12-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Luki.C4.87Lab17_9-0\" class=\"reference\"><a href=\"#cite_note-Luki.C4.87Lab17-9\">[9]<\/a><\/sup> This actual organization requires developing the clear-cut concepts of clinical-laboratory liaison and diagnostic stewardship, according to which the laboratory shall be engaged in reorganizing its structure (instrumentation, tests, staff) for providing an effective technical and advisory support to the local clinical needs within the network, whilst clinicians shall fairly cooperate with the laboratory staff for identifying the most efficient and effective solutions according to location and resources availability.<sup id=\"rdp-ebb-cite_ref-PlebaniDriving19_10-0\" class=\"reference\"><a href=\"#cite_note-PlebaniDriving19-10\">[10]<\/a><\/sup> An optimal balance should hence be always identified between case-mix (i.e., clinical complexity) of the healthcare facility where the laboratory is set and the locally available panel of diagnostic tests.\n<\/p><p>As an example, a regional reference center for management of severe bleeding disorders shall be equipped with a clinical laboratory capable to perform second-line and even third-line hemostasis tests, whilst a peripheral hospital within the same network would only need a basic hemostasis laboratory, since patients with severe bleeding disorder, either congenital or acquired, would be generally admitted and managed elsewhere.<sup id=\"rdp-ebb-cite_ref-LippiLabHemo18_11-0\" class=\"reference\"><a href=\"#cite_note-LippiLabHemo18-11\">[11]<\/a><\/sup> In the case that a patient with a bleeding syndrome is brought to a peripheral facility, the local \"spoke\" laboratory can then support clinicians with a panel of first-line (screening) hemostasis tests (e.g., prothrombin time, activated partial thromboplastin time, fibrinogen, platelet count, screening of platelet function). In most cases these tests\u2014along with the clinical history, signs, and symptoms\u2014will be sufficient to guide clinical decision making and the decision as to whether the patient may need to be referred to the reference center (where the \"hub\" laboratory is available) for being further investigated and eventually managed, or can else be locally treated or safely discharged. This paradigmatic example can then be translated to the vast majority of laboratory medicine areas (e.g., <a href=\"https:\/\/www.limswiki.org\/index.php\/Hematology\" title=\"Hematology\" class=\"wiki-link\" data-key=\"de8b49c7b0be3cec33af362e763b9b0c\">hematology<\/a>, immunochemistry, microbiology), by defining a clear hierarchy of tests that should be available in the different laboratories operating within a network. Regardless of personal inclinations and interests, \"spoke\" laboratories will generally need to be equipped with basic (first-line) laboratory tests, whereas \"hub\" laboratories will require more complex, time-consuming, and expensive (second- and third-line) analyses. As previously discussed, decisions on the final organization of laboratory diagnostics within a network of laboratory services must be taken in accordance with clinicians and hospital administrators, thus fulfilling clinical needs<sup id=\"rdp-ebb-cite_ref-J.C3.B8rgensenLeader17_12-0\" class=\"reference\"><a href=\"#cite_note-J.C3.B8rgensenLeader17-12\">[12]<\/a><\/sup>, principles of cost-effectiveness<sup id=\"rdp-ebb-cite_ref-LippiCost18_13-0\" class=\"reference\"><a href=\"#cite_note-LippiCost18-13\">[13]<\/a><\/sup>, and pre-analytical requirements.<sup id=\"rdp-ebb-cite_ref-LippiNovel17_14-0\" class=\"reference\"><a href=\"#cite_note-LippiNovel17-14\">[14]<\/a><\/sup> Dissipating both human and economic resources for performing obsolete, redundant, clinically questionable, or potentially unreliable analyses would not be beneficial for the healthcare system as a whole.\n<\/p><p>An accurate plan of technical resources will hence encompass a thorough analysis of the local situation, which will then influence the design of laboratory layout, preferably driven by validated tools such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Lean_laboratory\" title=\"Lean laboratory\" class=\"wiki-link\" data-key=\"6adad97006cb4eb4cac7dfcf767e3d5c\">Lean management<\/a> systems, which contextually enable to maximize efficiency and create a culture of <a href=\"https:\/\/www.limswiki.org\/index.php\/Continual_improvement_process\" title=\"Continual improvement process\" class=\"wiki-link\" data-key=\"fd7b54be3f6cdd0e8ed84d501486d668\">continuous improvement<\/a>.<sup id=\"rdp-ebb-cite_ref-KnowlesLean13_15-0\" class=\"reference\"><a href=\"#cite_note-KnowlesLean13-15\">[15]<\/a><\/sup> The leading factors associated with this process are volume and complexity (i.e., capacity), equipment and utility lists, staffing model, work schedule, <a href=\"https:\/\/www.limswiki.org\/index.php\/Regulatory_compliance\" title=\"Regulatory compliance\" class=\"wiki-link\" data-key=\"7dbc9be278a8efda25a4b592ee6ef0ca\">regulatory considerations<\/a>, safety ergonomic requirements, and, given space availability, research and education considerations (i.e., in academic centers).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Step_3_-_Staff_availability_and_qualification\">Step 3 - Staff availability and qualification<\/span><\/h2>\n<p>Whether this third step should follow or anticipate the planning of technical resources remains debated. This is mostly due to a recent metamorphosis that has occurred in staff availability. Personnel requirements have basically evolved from a demand conditioned by workflow, complexity and environment to a new scenario where shortage of public healthcare funding has contributed to make environment and staff availability (and qualification) the leading drivers of workflow and complexity. In 2014, a statistics of the World Health Organization (WHO) has alarmingly highlighted that the global shortage of doctors, nurses, midwives, and other healthcare professionals had already reached a 4.3 million deficit around the world.<sup id=\"rdp-ebb-cite_ref-AluttisTheWork14_16-0\" class=\"reference\"><a href=\"#cite_note-AluttisTheWork14-16\">[16]<\/a><\/sup> The situation has worsened in recent years, so that the predicted worldwide shortage of health care workers will probably exceed 15 million by the year 2030.<sup id=\"rdp-ebb-cite_ref-LiuGlobal17_17-0\" class=\"reference\"><a href=\"#cite_note-LiuGlobal17-17\">[17]<\/a><\/sup> Laboratory medicine makes no exception to this rule, since inefficient turnover has involved almost each category of laboratory professionals, especially during the past decade.<sup id=\"rdp-ebb-cite_ref-Cortelyou-WardTheLab11_18-0\" class=\"reference\"><a href=\"#cite_note-Cortelyou-WardTheLab11-18\">[18]<\/a><\/sup> It is hence rather understandable that laboratory managers must place staff availability among the top list of drivers of project management. Rome wasn\u2019t built in a day, though it would have never been built without a huge and skilled Roman workforce. \n<\/p><p><i>Mutatis mutandis<\/i>, volume, and complexity of laboratory testing must accurately be commensurate to the local availability of staff and to specific personnel education and qualifications. Importantly, when the available human resources do not meet predefined requirements of workforce and skills, additional strategies are required. These additional strategies must address further elimination of manual activities, automation of additional parts of the total testing process, and expanded consolidation of many different diagnostic areas, including the worst case scenario of reducing volume and complexity of diagnostic testing, or even outsourcing tests to private facilities.<sup id=\"rdp-ebb-cite_ref-LippiCost18_13-1\" class=\"reference\"><a href=\"#cite_note-LippiCost18-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LippiTheAdd17_19-0\" class=\"reference\"><a href=\"#cite_note-LippiTheAdd17-19\">[19]<\/a><\/sup> Among the possible solutions, this last option has recently gained significant momentum and has become especially appealing for some healthcare administrators, who are seeking to save money by cutting down laboratory funding and externalizing large volumes of tests. Whether or not this strategy is cost-effective remains largely disputed, although recently published evidence attests that outsourcing laboratory services decreases sample quality, increases turnaround time, and enhances the overall risk of diagnostic errors.<sup id=\"rdp-ebb-cite_ref-ChasinMedic07_20-0\" class=\"reference\"><a href=\"#cite_note-ChasinMedic07-20\">[20]<\/a><\/sup> Sizeable privatization of diagnostic testing is neither a clear-cut solution to the problem, since no reliable evidence has been provided that this will generate improved services and overall cost savings. Moreover, many doubts have been raised around the fact that private contractors do not need to openly disclose how public health money is spent, allocated, or collected. The meteoric ascent of Theranos in the firmament of laboratory medicine, followed by its rapid downfall, has taught us to be cautious in moving towards certain types of deregulated diagnostic testing.<sup id=\"rdp-ebb-cite_ref-LacknerTheTheran16_21-0\" class=\"reference\"><a href=\"#cite_note-LacknerTheTheran16-21\">[21]<\/a><\/sup>\n<\/p><p>Notably, critical issues in staff regulations (e.g., time on turn, recovery) must be clearly identified, and staff necessity should then be defined accordingly. A final consideration concerning personnel is that laboratory directors cannot usually select the staff. As such, they are rather constrained to develop leadership skills which will enable them to manage the existing personnel, thus placing the right person in the right place for doing the right activity at the right time. This obviously entails accurately knowing the potential personnel (i.e., weakness and strengths), trying to fulfill personal inclinations (whenever possible), and, especially, not blaming people when something goes wrong, since errors are frequently caused by a system failure rather than by individual mistakes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Step_4_-_Interplay_with_hospital_administration\">Step 4 - Interplay with hospital administration<\/span><\/h2>\n<p>As already anticipated in some previous parts of this article, laboratory professionals are increasingly involved in administrative duties, mostly encompassing test menu optimization, delivering training or education, and administering budgets.<sup id=\"rdp-ebb-cite_ref-PlebaniAMani19_22-0\" class=\"reference\"><a href=\"#cite_note-PlebaniAMani19-22\">[22]<\/a><\/sup> It is increasingly essential that laboratory directors and managers have a profound understanding of the budget of their laboratory and use that information for developing appropriate strategies for responding to a clinical demand, learning to manage budgets on the basis of a cost model, and presenting enough details to meet the needs of financial managers. These many aspects have become virtually unavoidable because laboratory diagnostics is now assimilated to many other economic industries by policymakers and administrators and is hence subjected to scale economy and evaluated accordingly. To put it simply, laboratory managers should aim to establish a favorable and constructive interplay with hospital administrators. Laboratory managers will also need to learn the language of hospital administrators and policymakers, since it is highly unlikely that these two categories will be ever strongly committed to speak a clinical language.<sup id=\"rdp-ebb-cite_ref-HorvathFromEvi13_23-0\" class=\"reference\"><a href=\"#cite_note-HorvathFromEvi13-23\">[23]<\/a><\/sup>\n<\/p><p>Regardless of any local organization, however, it is now undeniable that the future of laboratory medicine will be mostly driven by national healthcare policies, which are typically defined by a number of paradigms such as the amount of public funding for <i>in vitro<\/i> diagnostic testing, health insurance strategies, and test reimbursement policies. On a local basis, it will become increasingly essential to define medium- and long-term trajectories with hospital administrators, especially in terms of reorganization of the healthcare network (which will then influence number and size of laboratory services), number of hospital beds and outpatient flow (which will then influence test volume), and case-mix evolution (which will then influence test menu). Knowing this information in advance is necessary for developing an efficient and effective project management plan in laboratory medicine.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Several lines of evidence now attest that role and responsibilities of laboratory managers have considerably evolved over the last few decades.<sup id=\"rdp-ebb-cite_ref-JassamTheEuro18_24-0\" class=\"reference\"><a href=\"#cite_note-JassamTheEuro18-24\">[24]<\/a><\/sup> These paradigm shifts have been mostly driven by clinical, technical, economic, and social factors, mainly encompassing deepened understanding of the pathophysiology of human diseases, technological innovations, renewed focus on patient safety, cost-containment strategies, and patient empowerment. \n<\/p><p>The most obvious consequence has been the development of an ongoing process of reorganization, consolidation, and automation of laboratory services. The effective realization of those services requires defining an efficient project management plan, as well as constructing a new consciousness and an innovative and multifaceted job description of laboratory professionals worldwide. Notably, some other important drivers and outcome measures shall be considered when restructuring or redesigning the layout of a laboratory service, as briefly summarized in Table 2. These essentially include the identification and management of potential political or ideological resistances to the changes; the need to share the strategic plan with laboratory staff, local authorities, syndicates, and stakeholders (i.e., clinicians and patients); the definition of reliable performance indicators (both qualitative and quantitative), which will help assessing as to whether the new project is efficient and effective; and continuous monitoring of staff and stakeholder satisfaction. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 2.<\/b> Leading drivers of project management in laboratory medicine\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Step 1 - Defining the environment\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>a. Define space availability and organization.<br \/>\nb. Identify the most suitable local model of laboratory automation.\n<\/p>\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Step 2 \u2013 Planning technical resources\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>a. Define the geographical context (\"hub\" or \"spoke\" laboratory).<br \/>\nb. Recognize local clinical educational and research needs.<br \/>\nc. Identify predicted volume and complexity of tests.<br \/>\nd. Develop clinical-laboratory liaison and diagnostic stewardship.<br \/>\ne. Take advantage of using a Lean management system.\n<\/p>\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Step 3 \u2013 Staff availability and qualification\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>a. Define the number of available persons.<br \/>\nb. Acknowledge technical and clinical qualification of the staff.<br \/>\nc. Adapt volume and complexity of testing to available staff.<br \/>\nd. Identify critical issues in staff regulations (e.g., time on turn, recovery).<br \/>\ne. Place the right person in the right place for doing the right activity at the right time.<br \/>\nf. Safeguard personnel and patient safety.\n<\/p>\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Step 4 \u2013 Interplay with hospital administration\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>a. Seek out a constructive dialogue with hospital administrators.<br \/>\nb. Acknowledge the local political context.<br \/>\nc. Be aware of local healthcare plans (e.g., reorganization of healthcare network, number of beds, evolution of case-mix).<br \/>\nd. Become familiar with administrative duties and budgeting processes.\n<\/p>\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Step 5 \u2013 Additional drivers\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>a. Identify political or ideological resistances to the changes.<br \/>\nb. Share the strategic plan with laboratory staff, local authorities, syndicates, and stakeholders (i.e., clinicians and patients).<br \/>\nc. Define reliable performance indicators (qualitative and quantitative).<br \/>\nd. Continuously monitor staff and stakeholders satisfaction.<br \/>\ne. Identify an alternative solution (i.e., a backup plan).<br \/>\nf. Publicize results (when successful).\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Finally, additional consideration must be made. For example, common experience teaches that the delineation of an alternative solution\u2014should the primary solution not come to fruition\u2014may certainly be helpful to overcome possible technical failures of a new project. Whenever possible, the switch from the old to the new laboratory layout\u2014especially when entailing the use of novel instrumentation\u2014should not be irreversible. Additionally, the two solutions should be allowed to run in parallel for a certain period of time, at least until most of the possible problems have been identified and solved. Last but not least, provided that the final project proves successful, results must be publicized so that others may take profit from the local translation of favorable outcomes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This article is a detailed summary of an oral presentation delivered during the 15th Belgrade Symposium for Balkan Region, Belgrade (Serbia), 11\u201312 April, 2019.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors state that they have no conflicts of interest regarding the publication of this article.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-LippiTheIrr19-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LippiTheIrr19_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lippi, G. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6416815\" target=\"_blank\">\"The Irreplaceable Value of Laboratory Diagnostics: Four Recent Tests that have Revolutionized Clinical Practice\"<\/a>. <i>EJIFCC<\/i> <b>30<\/b> (1): 7\u201313. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6416815\/\" target=\"_blank\">PMC6416815<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30881270\" target=\"_blank\">30881270<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6416815\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6416815<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Irreplaceable+Value+of+Laboratory+Diagnostics%3A+Four+Recent+Tests+that+have+Revolutionized+Clinical+Practice&rft.jtitle=EJIFCC&rft.aulast=Lippi%2C+G.&rft.au=Lippi%2C+G.&rft.date=2019&rft.volume=30&rft.issue=1&rft.pages=7%E2%80%9313&rft_id=info:pmc\/PMC6416815&rft_id=info:pmid\/30881270&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6416815&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LippiWeigh18-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LippiWeigh18_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lippi, G. (2018). \"Weighting healthcare efficiency against available resources: value is the goal\". <i>Diagnosis<\/i> <b>5<\/b> (2): 39\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fdx-2018-0031\" target=\"_blank\">10.1515\/dx-2018-0031<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29858902\" target=\"_blank\">29858902<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Weighting+healthcare+efficiency+against+available+resources%3A+value+is+the+goal&rft.jtitle=Diagnosis&rft.aulast=Lippi%2C+G.&rft.au=Lippi%2C+G.&rft.date=2018&rft.volume=5&rft.issue=2&rft.pages=39%E2%80%9340&rft_id=info:doi\/10.1515%2Fdx-2018-0031&rft_id=info:pmid\/29858902&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Majki.C4.87-SinghLab17-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Majki.C4.87-SinghLab17_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Majki\u0107-Singh, N. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287220\" target=\"_blank\">\"Laboratory Medicine Management: Leadership Skills for Effective Laboratory\"<\/a>. <i>Journal of Medical Biochemistry<\/i> <b>36<\/b> (3): 207\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fjomb-2017-0034\" target=\"_blank\">10.1515\/jomb-2017-0034<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6287220\/\" target=\"_blank\">PMC6287220<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30564056\" target=\"_blank\">30564056<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287220\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287220<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+Medicine+Management%3A+Leadership+Skills+for+Effective+Laboratory&rft.jtitle=Journal+of+Medical+Biochemistry&rft.aulast=Majki%C4%87-Singh%2C+N.&rft.au=Majki%C4%87-Singh%2C+N.&rft.date=2017&rft.volume=36&rft.issue=3&rft.pages=207%E2%80%9310&rft_id=info:doi\/10.1515%2Fjomb-2017-0034&rft_id=info:pmc\/PMC6287220&rft_id=info:pmid\/30564056&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6287220&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PriceTrans18-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PriceTrans18_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Price, C.P.; Wolstenholme, J.; McGinley, P. et al. (2018). \"Translational health economics: The key to accountable adoption of in vitro diagnostic technologies\". <i>Health Services Management Research<\/i> <b>31<\/b> (1): 43\u201350. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F0951484817736727\" target=\"_blank\">10.1177\/0951484817736727<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29084478\" target=\"_blank\">29084478<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Translational+health+economics%3A+The+key+to+accountable+adoption+of+in+vitro+diagnostic+technologies&rft.jtitle=Health+Services+Management+Research&rft.aulast=Price%2C+C.P.%3B+Wolstenholme%2C+J.%3B+McGinley%2C+P.+et+al.&rft.au=Price%2C+C.P.%3B+Wolstenholme%2C+J.%3B+McGinley%2C+P.+et+al.&rft.date=2018&rft.volume=31&rft.issue=1&rft.pages=43%E2%80%9350&rft_id=info:doi\/10.1177%2F0951484817736727&rft_id=info:pmid\/29084478&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HawkerNon17-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HawkerNon17_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hawker, C.D. (2017). \"Nonanalytic Laboratory Automation: A Quarter Century of Progress\". <i>Clinical Chemistry<\/i> <b>63<\/b> (6): 1074-1082. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1373%2Fclinchem.2017.272047\" target=\"_blank\">10.1373\/clinchem.2017.272047<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28396562\" target=\"_blank\">28396562<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Nonanalytic+Laboratory+Automation%3A+A+Quarter+Century+of+Progress&rft.jtitle=Clinical+Chemistry&rft.aulast=Hawker%2C+C.D.&rft.au=Hawker%2C+C.D.&rft.date=2017&rft.volume=63&rft.issue=6&rft.pages=1074-1082&rft_id=info:doi\/10.1373%2Fclinchem.2017.272047&rft_id=info:pmid\/28396562&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GenzenChall18-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GenzenChall18_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Genzen, J.R.; Burnham, C.D.; Felder, R.A. et al. (2018). \"Challenges and Opportunities in Implementing Total Laboratory Automation\". <i>Clinical Chemistry<\/i> <b>64<\/b> (2): 259\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1373%2Fclinchem.2017.274068\" target=\"_blank\">10.1373\/clinchem.2017.274068<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28971983\" target=\"_blank\">28971983<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Challenges+and+Opportunities+in+Implementing+Total+Laboratory+Automation&rft.jtitle=Clinical+Chemistry&rft.aulast=Genzen%2C+J.R.%3B+Burnham%2C+C.D.%3B+Felder%2C+R.A.+et+al.&rft.au=Genzen%2C+J.R.%3B+Burnham%2C+C.D.%3B+Felder%2C+R.A.+et+al.&rft.date=2018&rft.volume=64&rft.issue=2&rft.pages=259%E2%80%9364&rft_id=info:doi\/10.1373%2Fclinchem.2017.274068&rft_id=info:pmid\/28971983&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LippiAdvant19-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LippiAdvant19_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lippi, G.; Da Rin, G. (2019). \"Advantages and limitations of total laboratory automation: A personal overview\". <i>Clinical Chemistry and Laboratory Medicine<\/i> <b>57<\/b> (6): 802\u201311. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fcclm-2018-1323\" target=\"_blank\">10.1515\/cclm-2018-1323<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30710480\" target=\"_blank\">30710480<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Advantages+and+limitations+of+total+laboratory+automation%3A+A+personal+overview&rft.jtitle=Clinical+Chemistry+and+Laboratory+Medicine&rft.aulast=Lippi%2C+G.%3B+Da+Rin%2C+G.&rft.au=Lippi%2C+G.%3B+Da+Rin%2C+G.&rft.date=2019&rft.volume=57&rft.issue=6&rft.pages=802%E2%80%9311&rft_id=info:doi\/10.1515%2Fcclm-2018-1323&rft_id=info:pmid\/30710480&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LippiLab12-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LippiLab12_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lippi, G.; Simundic, A.M. (2012). \"Laboratory networking and sample quality: A still relevant issue for patient safety\". <i>Clinical Chemistry and Laboratory Medicine<\/i> <b>50<\/b> (10): 1703\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fcclm-2012-0245\" target=\"_blank\">10.1515\/cclm-2012-0245<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23089698\" target=\"_blank\">23089698<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+networking+and+sample+quality%3A+A+still+relevant+issue+for+patient+safety&rft.jtitle=Clinical+Chemistry+and+Laboratory+Medicine&rft.aulast=Lippi%2C+G.%3B+Simundic%2C+A.M.&rft.au=Lippi%2C+G.%3B+Simundic%2C+A.M.&rft.date=2012&rft.volume=50&rft.issue=10&rft.pages=1703%E2%80%935&rft_id=info:doi\/10.1515%2Fcclm-2012-0245&rft_id=info:pmid\/23089698&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Luki.C4.87Lab17-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Luki.C4.87Lab17_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Luki\u0107, V. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287214\" target=\"_blank\">\"Laboratory Information System - Where are we Today?\"<\/a>. <i>Journal of Medical Biochemistry<\/i> <b>36<\/b> (3): 220-224. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fjomb-2017-0021\" target=\"_blank\">10.1515\/jomb-2017-0021<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6287214\/\" target=\"_blank\">PMC6287214<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30564059\" target=\"_blank\">30564059<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287214\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287214<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+Information+System+-+Where+are+we+Today%3F&rft.jtitle=Journal+of+Medical+Biochemistry&rft.aulast=Luki%C4%87%2C+V.&rft.au=Luki%C4%87%2C+V.&rft.date=2017&rft.volume=36&rft.issue=3&rft.pages=220-224&rft_id=info:doi\/10.1515%2Fjomb-2017-0021&rft_id=info:pmc\/PMC6287214&rft_id=info:pmid\/30564059&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6287214&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PlebaniDriving19-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PlebaniDriving19_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Plebani, M.; Laposata, M;. Lippi, G. (2019). \"Driving the route of laboratory medicine: A manifesto for the future\". <i>Internal and Emergency Medicine<\/i> <b>14<\/b> (3): 337\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11739-019-02053-z\" target=\"_blank\">10.1007\/s11739-019-02053-z<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30783946\" target=\"_blank\">30783946<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Driving+the+route+of+laboratory+medicine%3A+A+manifesto+for+the+future&rft.jtitle=Internal+and+Emergency+Medicine&rft.aulast=Plebani%2C+M.%3B+Laposata%2C+M%3B.+Lippi%2C+G.&rft.au=Plebani%2C+M.%3B+Laposata%2C+M%3B.+Lippi%2C+G.&rft.date=2019&rft.volume=14&rft.issue=3&rft.pages=337%E2%80%9340&rft_id=info:doi\/10.1007%2Fs11739-019-02053-z&rft_id=info:pmid\/30783946&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LippiLabHemo18-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LippiLabHemo18_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lippi, G., Favaloro, E.J. (2018). \"Laboratory hemostasis: From biology to the bench\". <i>Clinical Chemistry and Laboratory Medicine<\/i> <b>56<\/b> (7): 1035\u201345. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fcclm-2017-1205\" target=\"_blank\">10.1515\/cclm-2017-1205<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29455188\" target=\"_blank\">29455188<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+hemostasis%3A+From+biology+to+the+bench&rft.jtitle=Clinical+Chemistry+and+Laboratory+Medicine&rft.aulast=Lippi%2C+G.%2C+Favaloro%2C+E.J.&rft.au=Lippi%2C+G.%2C+Favaloro%2C+E.J.&rft.date=2018&rft.volume=56&rft.issue=7&rft.pages=1035%E2%80%9345&rft_id=info:doi\/10.1515%2Fcclm-2017-1205&rft_id=info:pmid\/29455188&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-J.C3.B8rgensenLeader17-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-J.C3.B8rgensenLeader17_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">J\u00f8rgensen, P.E. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287210\" target=\"_blank\">\"Leadership and Management in Clinical Biochemistry\"<\/a>. <i>Journal of Medical Biochemistry<\/i> <b>36<\/b> (3): 216\u201319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fjomb-2017-0023\" target=\"_blank\">10.1515\/jomb-2017-0023<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6287210\/\" target=\"_blank\">PMC6287210<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30564058\" target=\"_blank\">30564058<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287210\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6287210<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Leadership+and+Management+in+Clinical+Biochemistry&rft.jtitle=Journal+of+Medical+Biochemistry&rft.aulast=J%C3%B8rgensen%2C+P.E.&rft.au=J%C3%B8rgensen%2C+P.E.&rft.date=2017&rft.volume=36&rft.issue=3&rft.pages=216%E2%80%9319&rft_id=info:doi\/10.1515%2Fjomb-2017-0023&rft_id=info:pmc\/PMC6287210&rft_id=info:pmid\/30564058&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6287210&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LippiCost18-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LippiCost18_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-LippiCost18_13-1\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lippi, G.; Plebani, M. (2018). \"Cost, profitability and value of laboratory diagnostics: In God we trust, all others bring data\". <i>Journal of Laboratory Medicine<\/i> <b>43<\/b> (1): 1\u20133. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Flabmed-2018-0321\" target=\"_blank\">10.1515\/labmed-2018-0321<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cost%2C+profitability+and+value+of+laboratory+diagnostics%3A+In+God+we+trust%2C+all+others+bring+data&rft.jtitle=Journal+of+Laboratory+Medicine&rft.aulast=Lippi%2C+G.%3B+Plebani%2C+M.&rft.au=Lippi%2C+G.%3B+Plebani%2C+M.&rft.date=2018&rft.volume=43&rft.issue=1&rft.pages=1%E2%80%933&rft_id=info:doi\/10.1515%2Flabmed-2018-0321&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LippiNovel17-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LippiNovel17_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lippi, G., Cadamuro, J. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6294089\" target=\"_blank\">\"Novel Opportunities for Improving the Quality of Preanalytical Phase. A Glimpse to the Future?\"<\/a>. <i>Journal of Medical Biochemistry<\/i> <b>36<\/b> (4): 293-300. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fjomb-2017-0029\" target=\"_blank\">10.1515\/jomb-2017-0029<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6294089\/\" target=\"_blank\">PMC6294089<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30581325\" target=\"_blank\">30581325<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6294089\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6294089<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Novel+Opportunities+for+Improving+the+Quality+of+Preanalytical+Phase.+A+Glimpse+to+the+Future%3F&rft.jtitle=Journal+of+Medical+Biochemistry&rft.aulast=Lippi%2C+G.%2C+Cadamuro%2C+J.&rft.au=Lippi%2C+G.%2C+Cadamuro%2C+J.&rft.date=2017&rft.volume=36&rft.issue=4&rft.pages=293-300&rft_id=info:doi\/10.1515%2Fjomb-2017-0029&rft_id=info:pmc\/PMC6294089&rft_id=info:pmid\/30581325&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6294089&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KnowlesLean13-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KnowlesLean13_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Knowles, S.; Barnes, I. (2013). \"Lean laboratories: Laboratory medicine needs to learn from other industries how to deliver more for less\". <i>Journal of Clinical Pathology<\/i> <b>66<\/b> (8): 635\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fjclinpath-2013-201624\" target=\"_blank\">10.1136\/jclinpath-2013-201624<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23681948\" target=\"_blank\">23681948<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Lean+laboratories%3A+Laboratory+medicine+needs+to+learn+from+other+industries+how+to+deliver+more+for+less&rft.jtitle=Journal+of+Clinical+Pathology&rft.aulast=Knowles%2C+S.%3B+Barnes%2C+I.&rft.au=Knowles%2C+S.%3B+Barnes%2C+I.&rft.date=2013&rft.volume=66&rft.issue=8&rft.pages=635%E2%80%937&rft_id=info:doi\/10.1136%2Fjclinpath-2013-201624&rft_id=info:pmid\/23681948&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AluttisTheWork14-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AluttisTheWork14_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aluttis, C.; Bishaw, T.; Frank, M.W. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3926986\" target=\"_blank\">\"The workforce for health in a globalized context--global shortages and international migration\"<\/a>. <i>Global Health Action<\/i> <b>7<\/b>: 23611. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3402%2Fgha.v7.23611\" target=\"_blank\">10.3402\/gha.v7.23611<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3926986\/\" target=\"_blank\">PMC3926986<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24560265\" target=\"_blank\">24560265<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3926986\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3926986<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+workforce+for+health+in+a+globalized+context--global+shortages+and+international+migration&rft.jtitle=Global+Health+Action&rft.aulast=Aluttis%2C+C.%3B+Bishaw%2C+T.%3B+Frank%2C+M.W.&rft.au=Aluttis%2C+C.%3B+Bishaw%2C+T.%3B+Frank%2C+M.W.&rft.date=2014&rft.volume=7&rft.pages=23611&rft_id=info:doi\/10.3402%2Fgha.v7.23611&rft_id=info:pmc\/PMC3926986&rft_id=info:pmid\/24560265&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3926986&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiuGlobal17-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiuGlobal17_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liu, J.X.; Goryakin, Y.; Maeda, A. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5291995\" target=\"_blank\">\"Global Health Workforce Labor Market Projections for 2030\"<\/a>. <i>Human Resources for Health<\/i> <b>15<\/b> (1): 11. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12960-017-0187-2\" target=\"_blank\">10.1186\/s12960-017-0187-2<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5291995\/\" target=\"_blank\">PMC5291995<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28159017\" target=\"_blank\">28159017<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5291995\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5291995<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Global+Health+Workforce+Labor+Market+Projections+for+2030&rft.jtitle=Human+Resources+for+Health&rft.aulast=Liu%2C+J.X.%3B+Goryakin%2C+Y.%3B+Maeda%2C+A.+et+al.&rft.au=Liu%2C+J.X.%3B+Goryakin%2C+Y.%3B+Maeda%2C+A.+et+al.&rft.date=2017&rft.volume=15&rft.issue=1&rft.pages=11&rft_id=info:doi\/10.1186%2Fs12960-017-0187-2&rft_id=info:pmc\/PMC5291995&rft_id=info:pmid\/28159017&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5291995&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Cortelyou-WardTheLab11-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Cortelyou-WardTheLab11_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cortelyou-Ward, K.; Ramirez, B.; Rotarius, T. (2011). \"The laboratory workforce shortage: A managerial perspective\". <i>The Health Care Manager<\/i> <b>30<\/b> (2): 148\u201355. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FHCM.0b013e318216f5df\" target=\"_blank\">10.1097\/HCM.0b013e318216f5df<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21537137\" target=\"_blank\">21537137<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+laboratory+workforce+shortage%3A+A+managerial+perspective&rft.jtitle=The+Health+Care+Manager&rft.aulast=Cortelyou-Ward%2C+K.%3B+Ramirez%2C+B.%3B+Rotarius%2C+T.&rft.au=Cortelyou-Ward%2C+K.%3B+Ramirez%2C+B.%3B+Rotarius%2C+T.&rft.date=2011&rft.volume=30&rft.issue=2&rft.pages=148%E2%80%9355&rft_id=info:doi\/10.1097%2FHCM.0b013e318216f5df&rft_id=info:pmid\/21537137&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LippiTheAdd17-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LippiTheAdd17_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lippin, G.; Plebani, M. (2017). \"The add value of laboratory diagnostics: The many reasons why decision-makers should actually care\". <i>Journal of Laboratory and Precision Medicine<\/i> <b>2<\/b> (12): 100. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.21037%2Fjlpm.2017.12.07\" target=\"_blank\">10.21037\/jlpm.2017.12.07<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+add+value+of+laboratory+diagnostics%3A+The+many+reasons+why+decision-makers+should+actually+care&rft.jtitle=Journal+of+Laboratory+and+Precision+Medicine&rft.aulast=Lippin%2C+G.%3B+Plebani%2C+M.&rft.au=Lippin%2C+G.%3B+Plebani%2C+M.&rft.date=2017&rft.volume=2&rft.issue=12&rft.pages=100&rft_id=info:doi\/10.21037%2Fjlpm.2017.12.07&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChasinMedic07-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChasinMedic07_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chasin, B.S.; Elliott, S.P.; Klotz, S.A. (2007). \"Medical errors arising from outsourcing laboratory and radiology services\". <i>American Journal of Medicine<\/i> <b>120<\/b> (9): e9-11. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.amjmed.2006.07.024\" target=\"_blank\">10.1016\/j.amjmed.2006.07.024<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17765055\" target=\"_blank\">17765055<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+errors+arising+from+outsourcing+laboratory+and+radiology+services&rft.jtitle=American+Journal+of+Medicine&rft.aulast=Chasin%2C+B.S.%3B+Elliott%2C+S.P.%3B+Klotz%2C+S.A.&rft.au=Chasin%2C+B.S.%3B+Elliott%2C+S.P.%3B+Klotz%2C+S.A.&rft.date=2007&rft.volume=120&rft.issue=9&rft.pages=e9-11&rft_id=info:doi\/10.1016%2Fj.amjmed.2006.07.024&rft_id=info:pmid\/17765055&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LacknerTheTheran16-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LacknerTheTheran16_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lackner, K.J.; Gillery, P.; Lippi, G. et al. (2016). \"The Theranos phenomenon, scientific transparency and freedom of speech\". <i>Clinical Chemistry and Laboratory Medicine<\/i> <b>54<\/b> (9): 1403-5. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fcclm-2016-0520\" target=\"_blank\">10.1515\/cclm-2016-0520<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27442369\" target=\"_blank\">27442369<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Theranos+phenomenon%2C+scientific+transparency+and+freedom+of+speech&rft.jtitle=Clinical+Chemistry+and+Laboratory+Medicine&rft.aulast=Lackner%2C+K.J.%3B+Gillery%2C+P.%3B+Lippi%2C+G.+et+al.&rft.au=Lackner%2C+K.J.%3B+Gillery%2C+P.%3B+Lippi%2C+G.+et+al.&rft.date=2016&rft.volume=54&rft.issue=9&rft.pages=1403-5&rft_id=info:doi\/10.1515%2Fcclm-2016-0520&rft_id=info:pmid\/27442369&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PlebaniAMani19-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PlebaniAMani19_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Plebani, M.; Laposata, M.; Lippi, G. (2019). \"A manifesto for the future of laboratory medicine professionals\". <i>Clinica Chimica Acta<\/i> <b>489<\/b>: 49\u201352. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cca.2018.11.021\" target=\"_blank\">10.1016\/j.cca.2018.11.021<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30445032\" target=\"_blank\">30445032<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+manifesto+for+the+future+of+laboratory+medicine+professionals&rft.jtitle=Clinica+Chimica+Acta&rft.aulast=Plebani%2C+M.%3B+Laposata%2C+M.%3B+Lippi%2C+G.&rft.au=Plebani%2C+M.%3B+Laposata%2C+M.%3B+Lippi%2C+G.&rft.date=2019&rft.volume=489&rft.pages=49%E2%80%9352&rft_id=info:doi\/10.1016%2Fj.cca.2018.11.021&rft_id=info:pmid\/30445032&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HorvathFromEvi13-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HorvathFromEvi13_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Horvath, A.R. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3799219\" target=\"_blank\">\"From evidence to best practice in laboratory medicine\"<\/a>. <i>Clinical Biochemist Reviews<\/i> <b>34<\/b> (2): 47\u201360. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3799219\/\" target=\"_blank\">PMC3799219<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24151341\" target=\"_blank\">24151341<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3799219\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3799219<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+evidence+to+best+practice+in+laboratory+medicine&rft.jtitle=Clinical+Biochemist+Reviews&rft.aulast=Horvath%2C+A.R.&rft.au=Horvath%2C+A.R.&rft.date=2013&rft.volume=34&rft.issue=2&rft.pages=47%E2%80%9360&rft_id=info:pmc\/PMC3799219&rft_id=info:pmid\/24151341&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3799219&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JassamTheEuro18-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JassamTheEuro18_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jassam, N.; Lake, J.; Dabrowska, M. et al. (2018). \"The European Federation of Clinical Chemistry and Laboratory Medicine syllabus for postgraduate education and training for Specialists in Laboratory Medicine: Version 5 - 2018\". <i>Clinical Chemistry and Laboratory Medicine<\/i> <b>56<\/b> (11): 1846\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fcclm-2018-0344\" target=\"_blank\">10.1515\/cclm-2018-0344<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29870392\" target=\"_blank\">29870392<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+European+Federation+of+Clinical+Chemistry+and+Laboratory+Medicine+syllabus+for+postgraduate+education+and+training+for+Specialists+in+Laboratory+Medicine%3A+Version+5+-+2018&rft.jtitle=Clinical+Chemistry+and+Laboratory+Medicine&rft.aulast=Jassam%2C+N.%3B+Lake%2C+J.%3B+Dabrowska%2C+M.+et+al.&rft.au=Jassam%2C+N.%3B+Lake%2C+J.%3B+Dabrowska%2C+M.+et+al.&rft.date=2018&rft.volume=56&rft.issue=11&rft.pages=1846%E2%80%9363&rft_id=info:doi\/10.1515%2Fcclm-2018-0344&rft_id=info:pmid\/29870392&rfr_id=info:sid\/en.wikipedia.org:Journal:Project_management_in_laboratory_medicine\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference. Otherwise, in accordance with the NoDerivatives portion of the original license, nothing else has been changed.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204354\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.428 seconds\nReal time usage: 0.454 seconds\nPreprocessor visited node count: 20493\/1000000\nPreprocessor generated node count: 30697\/1000000\nPost\u2010expand include size: 162568\/2097152 bytes\nTemplate argument size: 50587\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 424.548 1 - -total\n 85.96% 364.940 1 - Template:Reflist\n 76.65% 325.436 24 - Template:Cite_journal\n 73.71% 312.937 24 - Template:Citation\/core\n 11.81% 50.156 52 - Template:Citation\/identifier\n 9.08% 38.550 1 - Template:Infobox_journal_article\n 8.64% 36.696 1 - Template:Infobox\n 5.84% 24.780 80 - Template:Infobox\/row\n 4.39% 18.658 24 - Template:Citation\/make_link\n 4.24% 18.016 112 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11425-0!*!0!!en!*!* and timestamp 20200707204354 and revision id 38076\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Project_management_in_laboratory_medicine\">https:\/\/www.limswiki.org\/index.php\/Journal:Project_management_in_laboratory_medicine<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","9d43e21cd08ee50665979d490d94d5ef_images":[],"9d43e21cd08ee50665979d490d94d5ef_timestamp":1594154634,"c28779d8f7eda776acc43ead069c1ea8_type":"article","c28779d8f7eda776acc43ead069c1ea8_title":"Universal LIMS-based platform for the automated processing of cell-based assays (Schmieder et al. 2019)","c28779d8f7eda776acc43ead069c1ea8_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays","c28779d8f7eda776acc43ead069c1ea8_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Universal LIMS-based platform for the automated processing of cell-based assays\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nUniversal LIMS-based platform for the automated processing of cell-based assaysJournal\n \nCurrent Directions in Biomedical EngineeringAuthor(s)\n \nSchmieder, Florian; Polk, Christoph; Gottl\u00f6ber, Felix; Sch\u00f6ps, Patrick; Sonntag, Frank; Deuse, Ronny; Jede, Aline; Petzold, ThomasAuthor affiliation(s)\n \nFraunhofer Institute for Material and Beam Technology IWS, Qualitype GmbHPrimary contact\n \nEmail: florian dot schmieder at iws dot fraunhofer dot deYear published\n \n2019Volume and issue\n \n5(1)Page(s)\n \n437\u201340DOI\n \n10.1515\/cdbme-2019-0110ISSN\n \n2364-5504Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.degruyter.com\/view\/j\/cdbme.2019.5.issue-1\/cdbme-2019-0110\/cdbme-2019-0110.xmlDownload\n \nhttps:\/\/www.degruyter.com\/downloadpdf\/j\/cdbme.2019.5.issue-1\/cdbme-2019-0110\/cdbme-2019-0110.xml (PDF)\n\nContents\n\n1 Abstract \n2 Introduction: The challenge to automate cell-based assays \n3 The nephrotoxicity assay \n4 Putting LIMS and LES to work: An example \n\n4.1 Automation platform \n4.2 LIMS integration via LES \n4.3 SiLA - Standardization in Lab Automation \n\n\n5 Conclusion \n6 Acknowledgements \n\n6.1 Funding \n6.2 Conflict of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nNowadays, cell-based assays are an elementary tool for diagnostics, animal-free substance testing, and basic research. Depending on the application, the spectrum ranges from simple static cell cultures in microtiter plates to dynamic co-cultures in complex micro-physiological systems (organ-on-a-chip). Depending on the complexity of the assay, numerous working steps have to be performed and the data from different analysis systems have to be processed, combined, and documented. A universal platform has been developed for the automated handling of cell-based assays, which combines a laboratory information management system (LIMS) with a laboratory execution system (LES), a universal laboratory automation platform and established laboratory equipment. The LIMS handles the administration of all laboratory-relevant information, the planning, control, and monitoring of laboratory processes, as well as the direct and qualified processing of raw data. Using a kidney-on-a-chip system as an example, the realization of complex cell-based assays for the animal-free characterization of the toxicity of different antibiotics will be demonstrated. In the kidney-on-a-chip system, the artificial proximal tubular barrier was formed by seeding human immortalized proximal tubule cells (RPTEC) and human blood outgrowth endothelial cells (BOEC) on ThinCert membranes. Transepithelial electrical resistance (TEER) was measured daily to evaluate the barrier function of the cellular layers. Fluid handling and TEER measurements were performed using a laboratory automation platform that communicates directly with the LIMS. The LES supported laboratory assistants in executing the manual handling steps of the experiments.\nKeywords: LIMS, LES, SiLA, cell based assay, lab automation\n\nIntroduction: The challenge to automate cell-based assays \nCell-based assays are nowadays widely used in drug discovery[1], personalized medicine[2][3], and basic research. This is due to the fact that cell sources like immortalized cell lines and iPS derived cells[4] offer a new range of possibilities regarding the assay sensitivity and reproducibility. Moreover, these cell sources could be harvested and cultivated as donor-specific, resulting in a wide range of patient-specific assay options. As such, many clinical researchers are developing cell-based assays to predict drug-specific interactions in vitro. In many therapeutic schemes, interactions of drugs with kidney-specific cells of the tubular and\/or glomerular compartment, as well as the disruption of the kidney barriers formed by those cells, are of major interest.[5] During the past decade, many steps of assay management, including media exchange and online monitoring via imaging, have been automated using laboratory automation platforms.[6][7] Nevertheless, these assays contain a multitude of steps, including initial cell seeding, supplementation, and measurement, which are executed manually. This could lead to a multitude of problems, particularly if during all the steps data integrity is not adequately considered. Therefore, laboratory informatics is needed to translate the tools of information technology into practical science by managing electronic records to optimize laboratory operations. Multiple systems may fulfill these requirements, including laboratory information management systems (LIMS) and scientific data management systems (SDMS).[8] Based on a nephrotoxicity assay, we will show how show how using a LIMS to integrate manual and automated workflows and complex datasets of different sources solves those issues.\n\nThe nephrotoxicity assay \nTesting nephrotoxicity in vitro means to investigate malfunctions of the glomerular or tubular barrier using cell culture models of those. Therefore the cellular assay is based on ThinCert inserts containing barrier models. Because the majority of molecular transporters\u2014which play an important role in secretion and reabsorption of drugs\u2014are located in the proximal tubule, most toxic effects arise in that part of the kidney.[9] As such, toxicity testing is based on an in vitro model of the proximal tubule. The whole assay consists of three main processes, with several sub-processes and steps, shown in Figure 1. It always starts with the formation of the tubular barrier. This process contains several manual sub-processes like cell seeding, cell maintenance, and quality assurance. These sub-processes are divided into working steps, e.g., \"media exchange.\" The total time to form the barrier is about four to seven days. When the barrier is formed and characterized\u2014to ensure quality aspects\u2014the toxicity assay is ready to begin. Taking clinically relevant malfunctions during medication into account, several parameters including cell viability, barrier integrity, and metabolic analysis have to be investigated. To emulate in vivo physiological conditions, regarding pressure and shear stress, the ThinCert inserts are integrated into a micro-physiological system (MPS) called a ZEBRA-Chip.[10] Like during barrier formation, manual steps including assembly and loading of solutions are performed. Nevertheless, the execution of several working steps during cell maintenance and quality assurance is done by a laboratory automation platform and the MPS control unit. This means that, in contrast to barrier formation, sub-processes like media exchange and TEER measurement are run automatically. The last process of the assay is endpoint analysis. Caused by staining exposure times, this process takes one to three days and is performed manually. In general the whole assay consists of 10 sub-processes performed by laboratory assistants and three sub-processes that are performed by laboratory robots or the MPS control unit. The most crucial sub-process in laboratory automation is TEER measurement, as it is performed manually and is automatic within the same assay. Hence TEER measurement is used to show how automation and integration into a laboratory execution system (LES) is done.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Nephrotoxicity assay at a glance. The standard assay consists of three steps: barrier formation, toxicity assay, and end-point analysis. All steps are divided into sub-processes that are done manually (indicated by human icon) or by laboratory automation system (gear icon).\n\n\n\nPutting LIMS and LES to work: An example \nA LIMS is a software application used by laboratories to acquire, analyze, store, and monitor laboratory data. The LIMS improves work processes by increasing productivity, traceability, sample turnaround time, and data security in many laboratories, including those in the life sciences industry.[11][12] As requirements of laboratories change, the demand for advanced data management solutions continues to grow, resulting in the development of modern, flexible LIMS capable of integrating SDMS, LES, and other informatics solutions.[11] The LIMS is the backbone of a laboratory, supporting the entire operation. Therefore, it provides all information from a single source and eliminates communication gaps between laboratory assistants, scientists, and other stakeholders, while also complying with the regulatory requirements found in the life sciences. Other benefits include more efficient laboratory workflows, simple retrieval of laboratory data, accelerated validation processes, and quality assurance of data. Beyond the LIMS, an LES can also aid in the execution of laboratory workflows, including material and product release, as well as quality testing. For laboratory professionals, an LES replaces manual data tracking with electronic processes and automates the interaction with methods, instruments, and supplies in routine laboratory procedures. By automated execution, protocols are completed efficiently and in a compliant and controlled fashion. In the life sciences, an LES is ideally suited to support key procedures in quality control and laboratory testing. Thus, method execution, calibration of instruments, data acquisition from instruments, data exchange, and data storage, as well as data review and processing can be executed through the LES.\nThe following subsections provide an example of combining these technologies for the automated processing of cell-based assays.\n\nAutomation platform \nThe developed LES is based on Qualitype GmbH's Abetter LIMS platform. A two-level portal robotics platform (PRP) manufactured by GeSiM mbH was used for laboratory automation, specifically in regards to the handling of liquids, imaging, and the measurement of TEER.[6] Microtiter plates and kidney-on-a-chip systems were placed on a temperature-controlled holder plate located between the two bridges. A versatile tool handler was mounted to the upper bridge, which can also be equipped with various fluid handling systems, sensors (TEER-Electrode) and tools. Another tool handler was mounted to the lower bridge. Each well or kidney-on-a-chip system could then be maintained separately with different cell culture media as well as monitored by fluorescence imaging and TEER-measurement.\n\nLIMS integration via LES \nTo make use of the beneficial effects that are described above for automation of data handling and processing of the nephrotoxicity assay, a universal LIMS platform was developed. It combines a LIMS with a LES for the automated handling of cell-based assays, while also integrating a universal laboratory automation platform and established laboratory equipment. Figure 2 shows the management of all laboratory-relevant information within the described assay monitored by the LIMS, such as the planning, control, and monitoring of laboratory processes, as well as the direct and qualified processing of raw data. Thus, it induces the work steps according to the assay's protocol via the LES to the laboratory assistant or via portal robotic control (PRC) to the PRP. The LES guides the laboratory assistant through the manual work steps of the experiments by using an interactive graphical user interface. The data gained during the experiment, as well as the documentation, recorded by checkboxes in-between work steps by the LES, are returned to the LIMS.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. LES pathways and communication between LIMS and LES. The laboratory assistant is interacting with the LIMS via LES (left column). Automated processes are controlled by the portal robotics control (PRC) (right column).\n\n\n\nSiLA - Standardization in Lab Automation \nConforming with the data management procedures shown in Figure 2, a powerful but universal data management standard is needed to face the obstacles during integration and implementation of laboratory informatics solutions.[12] Associations such as SiLA (Standardization in Lab Automation) and the International Consortium for Innovation and Quality in Pharmaceutical Development (IQ Consortium) are making significant attempts to introduce new interfaces and data management standards to facilitate system integration.[13] At the moment, the lack of integration standards is a major challenge to the greater application of laboratory informatics in many laboratories. With no unified strategy, laboratories largely continue to follow conventional procedures, regardless of efficiency, connectivity and interoperability between systems. SiLA standards enable a modular and versatile system approach to solve this issue. In line with these efforts, the TEER Measurement System was integrated as shown in Figure 3.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Dataflow during TEER measurement. The manually recorded data (left column) are converted by a BeagleBone black (BBB) that acts as a SiLA-Box. Data of automated processes (right column) are converted to SiLA by portal robotics control (PRC).\n\n\n\nFor manual handling of TEER measurement, the laboratory assistant is guided by the LES. The TEER measurement device (EVOM) was connected to a BeagleBone black (BBB) that converts analog to digital signals and acts as a data converter to the SiLA standard. Afterwards, data are transferred to the LIMS. Automated processes by the PRP are controlled by a SiLA-ready computer that acts as as a PRC and integrates a LabJack A\/D converter for transformation of analog signals.\n\nConclusion \nThis work shows how a combination of LIMS and LES can be used to integrate various manual and automated laboratory processes of cell-based assays within one informatics solution. Standard analog laboratory devices, like the TEER system used here, can be integrated easily in such systems by using embedded computers or hardware that can already function as a SiLA converter. The combination of laboratory informatics, devices, and universal gateways therefore helps to improve quality of acquired data and reduces sample turnaround time.\n\nAcknowledgements \nFunding \nThis study was funded by the European Union (SAB project 100216519 \u2013 CeCuLab) and the Federal Ministry of Economics and Technology (BMWi) according to a decision of the German Federal Parliament (IGF project 19175 BR\/1 - ZEBRA). \n\nConflict of interest \nRonny Deuse, Aline Jede, and Thomas Petzold work at Qualitype GmbH, which develops and sells laboratory information management systems. All other authors state no conflict of interest.\n\nReferences \n\n\n\u2191 Hering, Y.; Berthier, A.; Duez, H. et al. (2018). \"Development and implementation of a cell-based assay to discover agonists of the nuclear receptor REV-ERB\u03b1\". Journal of Biological Methods 5 (3): e94. doi:10.14440\/jbm.2018.244. PMC PMC6706147. PMID 31453244. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6706147 .   \n\n\u2191 Blom, K.; Nygren, P.; Alvarsson, J. et al. (2016). \"Ex Vivo Assessment of Drug Activity in Patient Tumor Cells as a Basis for Tailored Cancer Therapy\". Journal of Laboratory Automation 21 (1): 178\u201387. doi:10.1177\/2211068215598117. PMID 26246423.   \n\n\u2191 Blom, K.; Nygren, P.; Larsson, R. et al. (2017). \"Predictive Value of Ex Vivo Chemosensitivity Assays for Individualized Cancer Chemotherapy: A Meta-Analysis\". SLAS Technology 22 (3): 306-314. doi:10.1177\/2472630316686297. PMID 28378608.   \n\n\u2191 Rowe, R.G.; Daley, G.Q. (2019). \"Induced pluripotent stem cells in disease modelling and drug discovery\". Nature Reviews Genetics 20 (7): 377-388. doi:10.1038\/s41576-019-0100-z. PMC PMC6584039. PMID 30737492. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6584039 .   \n\n\u2191 Schmieder, F.; Sradnick, J.; Hempel, M. et al. (28 September 2018). \"Emulating gentamycin induced nephrotoxicity in an artificial system of the renal proximal tubule\". ResearchGate. Fraunhofer Institute for Material and Beam Technology. doi:10.13140\/RG.2.2.12274.96960. https:\/\/www.researchgate.net\/publication\/328172169_Emulating_gentamycin_induced_nephrotoxicity_in_an_artificial_system_of_the_renal_proximal_tubule .   \n\n\u2191 6.0 6.1 Schmieder, F.; Polk, C.; Eger, R. et al. (2014). \"Robotic platform for the automated handling and monitoring of cell culture devices\". Proceedings of Bionection 2014. doi:10.13140\/RG.2.1.2468.4247.   \n\n\u2191 Prabhu, G.R.D.; Urban, P.L. (2017). \"The dawn of unmanned analytical laboratories\". TrAC Trends in Analytical Chemistry 88: 41\u201352. doi:10.1016\/j.trac.2016.12.011.   \n\n\u2191 Neubert, S.; G\u00f6de, B.; Gu, X. et al. (2017). \"Potential of Laboratory Execution Systems (LESs) to Simplify the Application of Business Process Management Systems (BPMSs) in Laboratory Automation\". SLAS Technology 22 (2): 206-216. doi:10.1177\/2211068216680331. PMID 27908978.   \n\n\u2191 Yin, J.; Wang, J. (2016). \"Renal drug transporters and their significance in drug-drug interactions\". Acta Pharmaceutica Sinica B 6 (5): 363-373. doi:10.1016\/j.apsb.2016.07.013. PMC PMC5045553. PMID 27709005. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5045553 .   \n\n\u2191 Schmieder, F.; F\u00f6rster, D.; Hempel, M. et al. (23 May 2018). \"Mimicking human physiology at Transwell-based barrier models of the proximal tubule \u2013 The ZEBRA-Chip\". ResearchGate. Fraunhofer Institute for Material and Beam Technology. doi:10.13140\/RG.2.2.27364.48002. https:\/\/www.researchgate.net\/publication\/325398524_Mimicking_human_physiology_at_Transwell-based_barrier_models_of_the_proximal_tubule_-_The_ZEBRA-Chip .   \n\n\u2191 11.0 11.1 Tagger, B. (n.d.). \"An Introduction and Guide to Successfully Implementing a LIMS (Laboratory Information Management System)\" (PDF). Computer Science Department, University of Wales. http:\/\/www0.cs.ucl.ac.uk\/staff\/B.Tagger\/LimsPaper.pdf . Retrieved 05 June 2019 .   \n\n\u2191 12.0 12.1 Paszko, C.; Turner, E. (2001). Laboratory Information Management Systems (2nd ed.). CRC Press. ISBN 9780824741412. http:\/\/books.google.com\/books?id=ycEqnzPl2lYC .   \n\n\u2191 B\u00e4r, H.; Hochstrasser, R.; Papenfub, B. (2012). \"SiLA: Basic standards for rapid integration in laboratory automation\". Journal of Laboratory Automation 17 (2): 86\u201395. doi:10.1177\/2211068211424550. PMID 22357556.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\">https:\/\/www.limswiki.org\/index.php\/Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 11 February 2020, at 00:18.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 293 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","c28779d8f7eda776acc43ead069c1ea8_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Universal LIMS-based platform for the automated processing of cell-based assays<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Nowadays, cell-based assays are an elementary tool for diagnostics, animal-free substance testing, and basic research. Depending on the application, the spectrum ranges from simple static cell cultures in microtiter plates to dynamic co-cultures in complex micro-physiological systems (organ-on-a-chip). Depending on the complexity of the assay, numerous working steps have to be performed and the data from different analysis systems have to be processed, combined, and documented. A universal platform has been developed for the automated handling of cell-based assays, which combines a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management system<\/a> (LIMS) with a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_execution_system\" title=\"Laboratory execution system\" class=\"wiki-link\" data-key=\"774bdcab852f4d09565f0486bfafc26a\">laboratory execution system<\/a> (LES), a universal <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_automation\" title=\"Laboratory automation\" class=\"wiki-link\" data-key=\"0061880849aeaca05f8aa27ae171f331\">laboratory automation<\/a> platform and established <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> equipment. The LIMS handles the administration of all laboratory-relevant <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, the planning, control, and monitoring of laboratory processes, as well as the direct and qualified processing of raw data. Using a kidney-on-a-chip system as an example, the realization of complex cell-based assays for the animal-free characterization of the toxicity of different antibiotics will be demonstrated. In the kidney-on-a-chip system, the artificial proximal tubular barrier was formed by seeding human immortalized proximal tubule cells (RPTEC) and human blood outgrowth endothelial cells (BOEC) on ThinCert membranes. Transepithelial electrical resistance (TEER) was measured daily to evaluate the barrier function of the cellular layers. Fluid handling and TEER measurements were performed using a laboratory automation platform that communicates directly with the LIMS. The LES supported laboratory assistants in executing the manual handling steps of the experiments.\n<\/p><p><b>Keywords<\/b>: LIMS, LES, SiLA, cell based assay, lab automation\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction:_The_challenge_to_automate_cell-based_assays\">Introduction: The challenge to automate cell-based assays<\/span><\/h2>\n<p>Cell-based assays are nowadays widely used in drug discovery<sup id=\"rdp-ebb-cite_ref-HeringDevelop18_1-0\" class=\"reference\"><a href=\"#cite_note-HeringDevelop18-1\">[1]<\/a><\/sup>, personalized medicine<sup id=\"rdp-ebb-cite_ref-BlomExVivo16_2-0\" class=\"reference\"><a href=\"#cite_note-BlomExVivo16-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BlomPredict17_3-0\" class=\"reference\"><a href=\"#cite_note-BlomPredict17-3\">[3]<\/a><\/sup>, and basic research. This is due to the fact that cell sources like immortalized cell lines and iPS derived cells<sup id=\"rdp-ebb-cite_ref-RoweInduced19_4-0\" class=\"reference\"><a href=\"#cite_note-RoweInduced19-4\">[4]<\/a><\/sup> offer a new range of possibilities regarding the assay sensitivity and reproducibility. Moreover, these cell sources could be harvested and cultivated as donor-specific, resulting in a wide range of patient-specific assay options. As such, many clinical researchers are developing cell-based assays to predict drug-specific interactions <i>in vitro<\/i>. In many therapeutic schemes, interactions of drugs with kidney-specific cells of the tubular and\/or glomerular compartment, as well as the disruption of the kidney barriers formed by those cells, are of major interest.<sup id=\"rdp-ebb-cite_ref-SchmiederEmulat18_5-0\" class=\"reference\"><a href=\"#cite_note-SchmiederEmulat18-5\">[5]<\/a><\/sup> During the past decade, many steps of assay management, including media exchange and online monitoring via imaging, have been automated using <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_automation\" title=\"Laboratory automation\" class=\"wiki-link\" data-key=\"0061880849aeaca05f8aa27ae171f331\">laboratory automation<\/a> platforms.<sup id=\"rdp-ebb-cite_ref-SchmiederRobotic14_6-0\" class=\"reference\"><a href=\"#cite_note-SchmiederRobotic14-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PrabhuTheDawn17_7-0\" class=\"reference\"><a href=\"#cite_note-PrabhuTheDawn17-7\">[7]<\/a><\/sup> Nevertheless, these assays contain a multitude of steps, including initial cell seeding, supplementation, and measurement, which are executed manually. This could lead to a multitude of problems, particularly if during all the steps <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">data integrity<\/a> is not adequately considered. Therefore, <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_informatics\" title=\"Laboratory informatics\" class=\"wiki-link\" data-key=\"00edfa43edcde538a695f6d429280301\">laboratory informatics<\/a> is needed to translate the tools of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> technology into practical science by managing electronic records to optimize <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> operations. Multiple systems may fulfill these requirements, including <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> (LIMS) and <a href=\"https:\/\/www.limswiki.org\/index.php\/Scientific_data_management_system\" title=\"Scientific data management system\" class=\"wiki-link\" data-key=\"9f38d322b743f578fef487b6f3d7c253\">scientific data management systems<\/a> (SDMS).<sup id=\"rdp-ebb-cite_ref-NeubertPotent17_8-0\" class=\"reference\"><a href=\"#cite_note-NeubertPotent17-8\">[8]<\/a><\/sup> Based on a nephrotoxicity assay, we will show how show how using a LIMS to integrate manual and automated workflows and complex datasets of different sources solves those issues.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_nephrotoxicity_assay\">The nephrotoxicity assay<\/span><\/h2>\n<p>Testing nephrotoxicity <i>in vitro<\/i> means to investigate malfunctions of the glomerular or tubular barrier using cell culture models of those. Therefore the cellular assay is based on ThinCert inserts containing barrier models. Because the majority of molecular transporters\u2014which play an important role in secretion and reabsorption of drugs\u2014are located in the proximal tubule, most toxic effects arise in that part of the kidney.<sup id=\"rdp-ebb-cite_ref-YinRenal16_9-0\" class=\"reference\"><a href=\"#cite_note-YinRenal16-9\">[9]<\/a><\/sup> As such, toxicity testing is based on an <i>in vitro<\/i> model of the proximal tubule. The whole assay consists of three main processes, with several sub-processes and steps, shown in Figure 1. It always starts with the formation of the tubular barrier. This process contains several manual sub-processes like cell seeding, cell maintenance, and quality assurance. These sub-processes are divided into working steps, e.g., \"media exchange.\" The total time to form the barrier is about four to seven days. When the barrier is formed and characterized\u2014to ensure quality aspects\u2014the toxicity assay is ready to begin. Taking clinically relevant malfunctions during medication into account, several parameters including cell viability, barrier integrity, and metabolic analysis have to be investigated. To emulate <i>in vivo<\/i> physiological conditions, regarding pressure and shear stress, the ThinCert inserts are integrated into a micro-physiological system (MPS) called a ZEBRA-Chip.<sup id=\"rdp-ebb-cite_ref-SchmiederMimic18_10-0\" class=\"reference\"><a href=\"#cite_note-SchmiederMimic18-10\">[10]<\/a><\/sup> Like during barrier formation, manual steps including assembly and loading of solutions are performed. Nevertheless, the execution of several working steps during cell maintenance and quality assurance is done by a laboratory automation platform and the MPS control unit. This means that, in contrast to barrier formation, sub-processes like media exchange and TEER measurement are run automatically. The last process of the assay is endpoint analysis. Caused by staining exposure times, this process takes one to three days and is performed manually. In general the whole assay consists of 10 sub-processes performed by laboratory assistants and three sub-processes that are performed by laboratory robots or the MPS control unit. The most crucial sub-process in laboratory automation is TEER measurement, as it is performed manually and is automatic within the same assay. Hence TEER measurement is used to show how automation and integration into a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_execution_system\" title=\"Laboratory execution system\" class=\"wiki-link\" data-key=\"774bdcab852f4d09565f0486bfafc26a\">laboratory execution system<\/a> (LES) is done.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Schmieder_CurrentDirBioEng2019_5-1.png\" class=\"image wiki-link\" data-key=\"033e82cc36652cf0dffd5664b6f7a99f\"><img alt=\"Fig1 Schmieder CurrentDirBioEng2019 5-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8f\/Fig1_Schmieder_CurrentDirBioEng2019_5-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Nephrotoxicity assay at a glance. The standard assay consists of three steps: barrier formation, toxicity assay, and end-point analysis. All steps are divided into sub-processes that are done manually (indicated by human icon) or by laboratory automation system (gear icon).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Putting_LIMS_and_LES_to_work:_An_example\">Putting LIMS and LES to work: An example<\/span><\/h2>\n<p>A LIMS is a software application used by laboratories to acquire, analyze, store, and monitor laboratory data. The LIMS improves work processes by increasing productivity, traceability, sample turnaround time, and data security in many laboratories, including those in the life sciences industry.<sup id=\"rdp-ebb-cite_ref-TaggerAnInto_11-0\" class=\"reference\"><a href=\"#cite_note-TaggerAnInto-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LIMSPaszko01_12-0\" class=\"reference\"><a href=\"#cite_note-LIMSPaszko01-12\">[12]<\/a><\/sup> As requirements of laboratories change, the demand for advanced <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a> solutions continues to grow, resulting in the development of modern, flexible LIMS capable of integrating SDMS, LES, and other informatics solutions.<sup id=\"rdp-ebb-cite_ref-TaggerAnInto_11-1\" class=\"reference\"><a href=\"#cite_note-TaggerAnInto-11\">[11]<\/a><\/sup> The LIMS is the backbone of a laboratory, supporting the entire operation. Therefore, it provides all information from a single source and eliminates communication gaps between laboratory assistants, scientists, and other stakeholders, while also complying with the regulatory requirements found in the life sciences. Other benefits include more efficient laboratory <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a>, simple retrieval of laboratory data, accelerated validation processes, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_assurance\" title=\"Quality assurance\" class=\"wiki-link\" data-key=\"2ede4490f0ea707b14456f44439c0984\">quality assurance<\/a> of data. Beyond the LIMS, an LES can also aid in the execution of laboratory workflows, including material and product release, as well as quality testing. For laboratory professionals, an LES replaces manual data tracking with electronic processes and automates the interaction with methods, instruments, and supplies in routine laboratory procedures. By automated execution, protocols are completed efficiently and in a compliant and controlled fashion. In the life sciences, an LES is ideally suited to support key procedures in quality control and laboratory testing. Thus, method execution, calibration of instruments, data acquisition from instruments, data exchange, and data storage, as well as data review and processing can be executed through the LES.\n<\/p><p>The following subsections provide an example of combining these technologies for the automated processing of cell-based assays.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Automation_platform\">Automation platform<\/span><\/h3>\n<p>The developed LES is based on <a href=\"https:\/\/www.limswiki.org\/index.php\/Qualitype_GmbH\" title=\"Qualitype GmbH\" class=\"wiki-link\" data-key=\"bde56294c6e50189d1a4ddf65224214f\">Qualitype GmbH<\/a>'s Abetter LIMS platform. A two-level portal robotics platform (PRP) manufactured by GeSiM mbH was used for laboratory automation, specifically in regards to the handling of liquids, imaging, and the measurement of TEER.<sup id=\"rdp-ebb-cite_ref-SchmiederRobotic14_6-1\" class=\"reference\"><a href=\"#cite_note-SchmiederRobotic14-6\">[6]<\/a><\/sup> Microtiter plates and kidney-on-a-chip systems were placed on a temperature-controlled holder plate located between the two bridges. A versatile tool handler was mounted to the upper bridge, which can also be equipped with various fluid handling systems, sensors (TEER-Electrode) and tools. Another tool handler was mounted to the lower bridge. Each well or kidney-on-a-chip system could then be maintained separately with different cell culture media as well as monitored by fluorescence imaging and TEER-measurement.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"LIMS_integration_via_LES\">LIMS integration via LES<\/span><\/h3>\n<p>To make use of the beneficial effects that are described above for automation of data handling and processing of the nephrotoxicity assay, a universal LIMS platform was developed. It combines a LIMS with a LES for the automated handling of cell-based assays, while also integrating a universal laboratory automation platform and established laboratory equipment. Figure 2 shows the management of all laboratory-relevant information within the described assay monitored by the LIMS, such as the planning, control, and monitoring of laboratory processes, as well as the direct and qualified processing of raw data. Thus, it induces the work steps according to the assay's protocol via the LES to the laboratory assistant or via portal robotic control (PRC) to the PRP. The LES guides the laboratory assistant through the manual work steps of the experiments by using an interactive graphical user interface. The data gained during the experiment, as well as the documentation, recorded by checkboxes in-between work steps by the LES, are returned to the LIMS.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Schmieder_CurrentDirBioEng2019_5-1.png\" class=\"image wiki-link\" data-key=\"7009ba178aa47c3b2063eda09c98cd32\"><img alt=\"Fig2 Schmieder CurrentDirBioEng2019 5-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f2\/Fig2_Schmieder_CurrentDirBioEng2019_5-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> LES pathways and communication between LIMS and LES. The laboratory assistant is interacting with the LIMS via LES (left column). Automated processes are controlled by the portal robotics control (PRC) (right column).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"SiLA_-_Standardization_in_Lab_Automation\">SiLA - Standardization in Lab Automation<\/span><\/h3>\n<p>Conforming with the data management procedures shown in Figure 2, a powerful but universal data management standard is needed to face the obstacles during integration and implementation of laboratory informatics solutions.<sup id=\"rdp-ebb-cite_ref-LIMSPaszko01_12-1\" class=\"reference\"><a href=\"#cite_note-LIMSPaszko01-12\">[12]<\/a><\/sup> Associations such as SiLA (Standardization in Lab Automation) and the International Consortium for Innovation and Quality in Pharmaceutical Development (IQ Consortium) are making significant attempts to introduce new interfaces and data management standards to facilitate system integration.<sup id=\"rdp-ebb-cite_ref-B.C3.A4rSiLA12_13-0\" class=\"reference\"><a href=\"#cite_note-B.C3.A4rSiLA12-13\">[13]<\/a><\/sup> At the moment, the lack of integration standards is a major challenge to the greater application of laboratory informatics in many laboratories. With no unified strategy, laboratories largely continue to follow conventional procedures, regardless of efficiency, connectivity and interoperability between systems. SiLA standards enable a modular and versatile system approach to solve this issue. In line with these efforts, the TEER Measurement System was integrated as shown in Figure 3.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Schmieder_CurrentDirBioEng2019_5-1.png\" class=\"image wiki-link\" data-key=\"d052b940a5dc9819755ec302af65dbf3\"><img alt=\"Fig3 Schmieder CurrentDirBioEng2019 5-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3b\/Fig3_Schmieder_CurrentDirBioEng2019_5-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Dataflow during TEER measurement. The manually recorded data (left column) are converted by a BeagleBone black (BBB) that acts as a SiLA-Box. Data of automated processes (right column) are converted to SiLA by portal robotics control (PRC).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For manual handling of TEER measurement, the laboratory assistant is guided by the LES. The TEER measurement device (EVOM) was connected to a BeagleBone black (BBB) that converts analog to digital signals and acts as a data converter to the SiLA standard. Afterwards, data are transferred to the LIMS. Automated processes by the PRP are controlled by a SiLA-ready computer that acts as as a PRC and integrates a LabJack A\/D converter for transformation of analog signals.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>This work shows how a combination of LIMS and LES can be used to integrate various manual and automated laboratory processes of cell-based assays within one informatics solution. Standard analog laboratory devices, like the TEER system used here, can be integrated easily in such systems by using embedded computers or hardware that can already function as a SiLA converter. The combination of laboratory informatics, devices, and universal gateways therefore helps to improve quality of acquired data and reduces sample turnaround time.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This study was funded by the European Union (SAB project 100216519 \u2013 CeCuLab) and the Federal Ministry of Economics and Technology (BMWi) according to a decision of the German Federal Parliament (IGF project 19175 BR\/1 - ZEBRA). \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>Ronny Deuse, Aline Jede, and Thomas Petzold work at Qualitype GmbH, which develops and sells laboratory information management systems. All other authors state no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HeringDevelop18-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeringDevelop18_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hering, Y.; Berthier, A.; Duez, H. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6706147\" target=\"_blank\">\"Development and implementation of a cell-based assay to discover agonists of the nuclear receptor REV-ERB\u03b1\"<\/a>. <i>Journal of Biological Methods<\/i> <b>5<\/b> (3): e94. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.14440%2Fjbm.2018.244\" target=\"_blank\">10.14440\/jbm.2018.244<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6706147\/\" target=\"_blank\">PMC6706147<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31453244\" target=\"_blank\">31453244<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6706147\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6706147<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+and+implementation+of+a+cell-based+assay+to+discover+agonists+of+the+nuclear+receptor+REV-ERB%CE%B1&rft.jtitle=Journal+of+Biological+Methods&rft.aulast=Hering%2C+Y.%3B+Berthier%2C+A.%3B+Duez%2C+H.+et+al.&rft.au=Hering%2C+Y.%3B+Berthier%2C+A.%3B+Duez%2C+H.+et+al.&rft.date=2018&rft.volume=5&rft.issue=3&rft.pages=e94&rft_id=info:doi\/10.14440%2Fjbm.2018.244&rft_id=info:pmc\/PMC6706147&rft_id=info:pmid\/31453244&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6706147&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlomExVivo16-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlomExVivo16_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blom, K.; Nygren, P.; Alvarsson, J. et al. (2016). \"Ex Vivo Assessment of Drug Activity in Patient Tumor Cells as a Basis for Tailored Cancer Therapy\". <i>Journal of Laboratory Automation<\/i> <b>21<\/b> (1): 178\u201387. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F2211068215598117\" target=\"_blank\">10.1177\/2211068215598117<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26246423\" target=\"_blank\">26246423<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ex+Vivo+Assessment+of+Drug+Activity+in+Patient+Tumor+Cells+as+a+Basis+for+Tailored+Cancer+Therapy&rft.jtitle=Journal+of+Laboratory+Automation&rft.aulast=Blom%2C+K.%3B+Nygren%2C+P.%3B+Alvarsson%2C+J.+et+al.&rft.au=Blom%2C+K.%3B+Nygren%2C+P.%3B+Alvarsson%2C+J.+et+al.&rft.date=2016&rft.volume=21&rft.issue=1&rft.pages=178%E2%80%9387&rft_id=info:doi\/10.1177%2F2211068215598117&rft_id=info:pmid\/26246423&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlomPredict17-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlomPredict17_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blom, K.; Nygren, P.; Larsson, R. et al. (2017). \"Predictive Value of Ex Vivo Chemosensitivity Assays for Individualized Cancer Chemotherapy: A Meta-Analysis\". <i>SLAS Technology<\/i> <b>22<\/b> (3): 306-314. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F2472630316686297\" target=\"_blank\">10.1177\/2472630316686297<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28378608\" target=\"_blank\">28378608<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Predictive+Value+of+Ex+Vivo+Chemosensitivity+Assays+for+Individualized+Cancer+Chemotherapy%3A+A+Meta-Analysis&rft.jtitle=SLAS+Technology&rft.aulast=Blom%2C+K.%3B+Nygren%2C+P.%3B+Larsson%2C+R.+et+al.&rft.au=Blom%2C+K.%3B+Nygren%2C+P.%3B+Larsson%2C+R.+et+al.&rft.date=2017&rft.volume=22&rft.issue=3&rft.pages=306-314&rft_id=info:doi\/10.1177%2F2472630316686297&rft_id=info:pmid\/28378608&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoweInduced19-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RoweInduced19_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rowe, R.G.; Daley, G.Q. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6584039\" target=\"_blank\">\"Induced pluripotent stem cells in disease modelling and drug discovery\"<\/a>. <i>Nature Reviews Genetics<\/i> <b>20<\/b> (7): 377-388. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41576-019-0100-z\" target=\"_blank\">10.1038\/s41576-019-0100-z<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6584039\/\" target=\"_blank\">PMC6584039<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30737492\" target=\"_blank\">30737492<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6584039\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6584039<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Induced+pluripotent+stem+cells+in+disease+modelling+and+drug+discovery&rft.jtitle=Nature+Reviews+Genetics&rft.aulast=Rowe%2C+R.G.%3B+Daley%2C+G.Q.&rft.au=Rowe%2C+R.G.%3B+Daley%2C+G.Q.&rft.date=2019&rft.volume=20&rft.issue=7&rft.pages=377-388&rft_id=info:doi\/10.1038%2Fs41576-019-0100-z&rft_id=info:pmc\/PMC6584039&rft_id=info:pmid\/30737492&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6584039&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchmiederEmulat18-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchmiederEmulat18_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Schmieder, F.; Sradnick, J.; Hempel, M. et al. (28 September 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchgate.net\/publication\/328172169_Emulating_gentamycin_induced_nephrotoxicity_in_an_artificial_system_of_the_renal_proximal_tubule\" target=\"_blank\">\"Emulating gentamycin induced nephrotoxicity in an artificial system of the renal proximal tubule\"<\/a>. <i>ResearchGate<\/i>. Fraunhofer Institute for Material and Beam Technology. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13140%2FRG.2.2.12274.96960\" target=\"_blank\">10.13140\/RG.2.2.12274.96960<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.researchgate.net\/publication\/328172169_Emulating_gentamycin_induced_nephrotoxicity_in_an_artificial_system_of_the_renal_proximal_tubule\" target=\"_blank\">https:\/\/www.researchgate.net\/publication\/328172169_Emulating_gentamycin_induced_nephrotoxicity_in_an_artificial_system_of_the_renal_proximal_tubule<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Emulating+gentamycin+induced+nephrotoxicity+in+an+artificial+system+of+the+renal+proximal+tubule&rft.atitle=ResearchGate&rft.aulast=Schmieder%2C+F.%3B+Sradnick%2C+J.%3B+Hempel%2C+M.+et+al.&rft.au=Schmieder%2C+F.%3B+Sradnick%2C+J.%3B+Hempel%2C+M.+et+al.&rft.date=28+September+2018&rft.pub=Fraunhofer+Institute+for+Material+and+Beam+Technology&rft_id=info:doi\/10.13140%2FRG.2.2.12274.96960&rft_id=https%3A%2F%2Fwww.researchgate.net%2Fpublication%2F328172169_Emulating_gentamycin_induced_nephrotoxicity_in_an_artificial_system_of_the_renal_proximal_tubule&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchmiederRobotic14-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SchmiederRobotic14_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-SchmiederRobotic14_6-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schmieder, F.; Polk, C.; Eger, R. et al. (2014). \"Robotic platform for the automated handling and monitoring of cell culture devices\". <i>Proceedings of Bionection 2014<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13140%2FRG.2.1.2468.4247\" target=\"_blank\">10.13140\/RG.2.1.2468.4247<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Robotic+platform+for+the+automated+handling+and+monitoring+of+cell+culture+devices&rft.jtitle=Proceedings+of+Bionection+2014&rft.aulast=Schmieder%2C+F.%3B+Polk%2C+C.%3B+Eger%2C+R.+et+al.&rft.au=Schmieder%2C+F.%3B+Polk%2C+C.%3B+Eger%2C+R.+et+al.&rft.date=2014&rft_id=info:doi\/10.13140%2FRG.2.1.2468.4247&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PrabhuTheDawn17-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PrabhuTheDawn17_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Prabhu, G.R.D.; Urban, P.L. (2017). \"The dawn of unmanned analytical laboratories\". <i>TrAC Trends in Analytical Chemistry<\/i> <b>88<\/b>: 41\u201352. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.trac.2016.12.011\" target=\"_blank\">10.1016\/j.trac.2016.12.011<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+dawn+of+unmanned+analytical+laboratories&rft.jtitle=TrAC+Trends+in+Analytical+Chemistry&rft.aulast=Prabhu%2C+G.R.D.%3B+Urban%2C+P.L.&rft.au=Prabhu%2C+G.R.D.%3B+Urban%2C+P.L.&rft.date=2017&rft.volume=88&rft.pages=41%E2%80%9352&rft_id=info:doi\/10.1016%2Fj.trac.2016.12.011&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NeubertPotent17-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NeubertPotent17_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Neubert, S.; G\u00f6de, B.; Gu, X. et al. (2017). \"Potential of Laboratory Execution Systems (LESs) to Simplify the Application of Business Process Management Systems (BPMSs) in Laboratory Automation\". <i>SLAS Technology<\/i> <b>22<\/b> (2): 206-216. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F2211068216680331\" target=\"_blank\">10.1177\/2211068216680331<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27908978\" target=\"_blank\">27908978<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Potential+of+Laboratory+Execution+Systems+%28LESs%29+to+Simplify+the+Application+of+Business+Process+Management+Systems+%28BPMSs%29+in+Laboratory+Automation&rft.jtitle=SLAS+Technology&rft.aulast=Neubert%2C+S.%3B+G%C3%B6de%2C+B.%3B+Gu%2C+X.+et+al.&rft.au=Neubert%2C+S.%3B+G%C3%B6de%2C+B.%3B+Gu%2C+X.+et+al.&rft.date=2017&rft.volume=22&rft.issue=2&rft.pages=206-216&rft_id=info:doi\/10.1177%2F2211068216680331&rft_id=info:pmid\/27908978&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YinRenal16-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YinRenal16_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yin, J.; Wang, J. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5045553\" target=\"_blank\">\"Renal drug transporters and their significance in drug-drug interactions\"<\/a>. <i>Acta Pharmaceutica Sinica B<\/i> <b>6<\/b> (5): 363-373. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.apsb.2016.07.013\" target=\"_blank\">10.1016\/j.apsb.2016.07.013<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5045553\/\" target=\"_blank\">PMC5045553<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27709005\" target=\"_blank\">27709005<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5045553\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5045553<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Renal+drug+transporters+and+their+significance+in+drug-drug+interactions&rft.jtitle=Acta+Pharmaceutica+Sinica+B&rft.aulast=Yin%2C+J.%3B+Wang%2C+J.&rft.au=Yin%2C+J.%3B+Wang%2C+J.&rft.date=2016&rft.volume=6&rft.issue=5&rft.pages=363-373&rft_id=info:doi\/10.1016%2Fj.apsb.2016.07.013&rft_id=info:pmc\/PMC5045553&rft_id=info:pmid\/27709005&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5045553&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchmiederMimic18-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchmiederMimic18_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Schmieder, F.; F\u00f6rster, D.; Hempel, M. et al. (23 May 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchgate.net\/publication\/325398524_Mimicking_human_physiology_at_Transwell-based_barrier_models_of_the_proximal_tubule_-_The_ZEBRA-Chip\" target=\"_blank\">\"Mimicking human physiology at Transwell-based barrier models of the proximal tubule \u2013 The ZEBRA-Chip\"<\/a>. <i>ResearchGate<\/i>. Fraunhofer Institute for Material and Beam Technology. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13140%2FRG.2.2.27364.48002\" target=\"_blank\">10.13140\/RG.2.2.27364.48002<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.researchgate.net\/publication\/325398524_Mimicking_human_physiology_at_Transwell-based_barrier_models_of_the_proximal_tubule_-_The_ZEBRA-Chip\" target=\"_blank\">https:\/\/www.researchgate.net\/publication\/325398524_Mimicking_human_physiology_at_Transwell-based_barrier_models_of_the_proximal_tubule_-_The_ZEBRA-Chip<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Mimicking+human+physiology+at+Transwell-based+barrier+models+of+the+proximal+tubule+%E2%80%93+The+ZEBRA-Chip&rft.atitle=ResearchGate&rft.aulast=Schmieder%2C+F.%3B+F%C3%B6rster%2C+D.%3B+Hempel%2C+M.+et+al.&rft.au=Schmieder%2C+F.%3B+F%C3%B6rster%2C+D.%3B+Hempel%2C+M.+et+al.&rft.date=23+May+2018&rft.pub=Fraunhofer+Institute+for+Material+and+Beam+Technology&rft_id=info:doi\/10.13140%2FRG.2.2.27364.48002&rft_id=https%3A%2F%2Fwww.researchgate.net%2Fpublication%2F325398524_Mimicking_human_physiology_at_Transwell-based_barrier_models_of_the_proximal_tubule_-_The_ZEBRA-Chip&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaggerAnInto-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TaggerAnInto_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-TaggerAnInto_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Tagger, B. (n.d.). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www0.cs.ucl.ac.uk\/staff\/B.Tagger\/LimsPaper.pdf\" target=\"_blank\">\"An Introduction and Guide to Successfully Implementing a LIMS (Laboratory Information Management System)\"<\/a> (PDF). Computer Science Department, University of Wales<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www0.cs.ucl.ac.uk\/staff\/B.Tagger\/LimsPaper.pdf\" target=\"_blank\">http:\/\/www0.cs.ucl.ac.uk\/staff\/B.Tagger\/LimsPaper.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 June 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=An+Introduction+and+Guide+to+Successfully+Implementing+a+LIMS+%28Laboratory+Information+Management+System%29&rft.atitle=&rft.aulast=Tagger%2C+B.&rft.au=Tagger%2C+B.&rft.date=n.d.&rft.pub=Computer+Science+Department%2C+University+of+Wales&rft_id=http%3A%2F%2Fwww0.cs.ucl.ac.uk%2Fstaff%2FB.Tagger%2FLimsPaper.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LIMSPaszko01-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LIMSPaszko01_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-LIMSPaszko01_12-1\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Paszko, C.; Turner, E. (2001). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/books.google.com\/books?id=ycEqnzPl2lYC\" target=\"_blank\"><i>Laboratory Information Management Systems<\/i><\/a> (2nd ed.). CRC Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780824741412<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/books.google.com\/books?id=ycEqnzPl2lYC\" target=\"_blank\">http:\/\/books.google.com\/books?id=ycEqnzPl2lYC<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Laboratory+Information+Management+Systems&rft.aulast=Paszko%2C+C.%3B+Turner%2C+E.&rft.au=Paszko%2C+C.%3B+Turner%2C+E.&rft.date=2001&rft.edition=2nd&rft.pub=CRC+Press&rft.isbn=9780824741412&rft_id=http%3A%2F%2Fbooks.google.com%2Fbooks%3Fid%3DycEqnzPl2lYC&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-B.C3.A4rSiLA12-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-B.C3.A4rSiLA12_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">B\u00e4r, H.; Hochstrasser, R.; Papenfub, B. (2012). \"SiLA: Basic standards for rapid integration in laboratory automation\". <i>Journal of Laboratory Automation<\/i> <b>17<\/b> (2): 86\u201395. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F2211068211424550\" target=\"_blank\">10.1177\/2211068211424550<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22357556\" target=\"_blank\">22357556<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SiLA%3A+Basic+standards+for+rapid+integration+in+laboratory+automation&rft.jtitle=Journal+of+Laboratory+Automation&rft.aulast=B%C3%A4r%2C+H.%3B+Hochstrasser%2C+R.%3B+Papenfub%2C+B.&rft.au=B%C3%A4r%2C+H.%3B+Hochstrasser%2C+R.%3B+Papenfub%2C+B.&rft.date=2012&rft.volume=17&rft.issue=2&rft.pages=86%E2%80%9395&rft_id=info:doi\/10.1177%2F2211068211424550&rft_id=info:pmid\/22357556&rfr_id=info:sid\/en.wikipedia.org:Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204354\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.329 seconds\nReal time usage: 2.012 seconds\nPreprocessor visited node count: 11648\/1000000\nPreprocessor generated node count: 33703\/1000000\nPost\u2010expand include size: 96252\/2097152 bytes\nTemplate argument size: 34161\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 252.272 1 - -total\n 78.74% 198.638 1 - Template:Reflist\n 64.61% 162.983 13 - Template:Citation\/core\n 50.25% 126.765 9 - Template:Cite_journal\n 15.46% 39.005 1 - Template:Infobox_journal_article\n 14.83% 37.408 1 - Template:Infobox\n 13.41% 33.839 3 - Template:Cite_web\n 10.15% 25.597 80 - Template:Infobox\/row\n 8.12% 20.495 22 - Template:Citation\/identifier\n 5.19% 13.090 1 - Template:Cite_book\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11412-0!*!0!!en!5!* and timestamp 20200707204352 and revision id 37949\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays\">https:\/\/www.limswiki.org\/index.php\/Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","c28779d8f7eda776acc43ead069c1ea8_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8f\/Fig1_Schmieder_CurrentDirBioEng2019_5-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f2\/Fig2_Schmieder_CurrentDirBioEng2019_5-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3b\/Fig3_Schmieder_CurrentDirBioEng2019_5-1.png"],"c28779d8f7eda776acc43ead069c1ea8_timestamp":1594154631,"eb76e424fc635068c27be185a754eac7_type":"article","eb76e424fc635068c27be185a754eac7_title":"Design and implementation of a clinical laboratory information system in a low-resource setting (Mtonga et al. 2019)","eb76e424fc635068c27be185a754eac7_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting","eb76e424fc635068c27be185a754eac7_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Design and implementation of a clinical laboratory information system in a low-resource setting\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDesign and implementation of a clinical laboratory information system in a low-resource settingJournal\n \nAfrican Journal of Laboratory MedicineAuthor(s)\n \nMtonga, Timothy M.; Choonara, Faheema E.; Espino, Jeremy U.; Kachaje, Chimwemwe; Kapundi, Kenneth;\r\nMengezi, Takondwa E.; Mumba, Soyapi L.; Douglas, Gerald P.Author affiliation(s)\n \nUniversity of Pittsburgh, Kamuzu Central Hospital, Baobab Health TrustPrimary contact\n \nEmail: tmm113 at pitt dot eduYear published\n \n2019Volume and issue\n \n8(1)Article #\n \na841DOI\n \n10.4102\/ajlm.v8i1.841ISSN\n \n2225-2010Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/ajlmonline.org\/index.php\/ajlm\/article\/view\/841\/1391Download\n \nNone\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methods \n\n3.1 Ethical considerations \n3.2 Setting \n3.3 User requirements and system capabilities \n3.4 System design and development \n\n\n4 Results \n\n4.1 User requirements and system capabilities \n4.2 System design and development \n4.3 Developing bedside solutions \n4.4 Laboratory testing workflow with the laboratory information system \n\n\n5 Discussion \n6 Acknowledgements \n\n6.1 Competing interests \n6.2 Authors\u2019 contributions \n6.3 Sources of support \n6.4 Data availability statement \n6.5 Disclaimer \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nBackground: Reducing laboratory errors presents a significant opportunity for both cost reduction and healthcare quality improvement. This is particularly true in low-resource settings where laboratory errors are further exacerbated by poor infrastructure and shortages in a trained workforce. informatics interventions can be used to address some of the sources of laboratory errors.\nObjectives: This article describes the development process for a clinical laboratory information system (LIS) that leverages informatics interventions to address problems in the laboratory testing process at a hospital in a low-resource setting.\nMethods: We designed interventions using informatics methods for previously identified problems in the laboratory testing process at a clinical laboratory in a low-resource setting. First, we reviewed a pre-existing LIS functionality assessment toolkit and consulted with laboratory personnel. This provided requirements that were developed into a LIS with interventions designed to address the problems that had been identified. We piloted the LIS at the Kamuzu Central Hospital in Lilongwe, Malawi.\nResults: We implemented a series of informatics interventions in the form of a LIS to address sources of laboratory errors and support the entire laboratory testing process. Custom hardware was built to support the ordering of laboratory tests and review of laboratory test results.\nConclusion: Our experience highlights the potential of using informatics interventions to address systemic problems in the laboratory testing process in low-resource settings. Implementing these interventions may require innovation of new hardware to address various contextual issues. We strongly encourage thorough testing of such innovations to reduce the risk of failure when implemented.\nKeywords: low-resource setting, laboratory testing, laboratory information system, Malawi, informatics interventions\n\nIntroduction \nLaboratory testing plays a vital role in clinical decision-making. It is estimated that up to 70% of medical decisions in high-resource healthcare settings are made based on clinical laboratory test results.[1][2] Even though access to clinical laboratory services is comparatively lower in low-resource settings, studies show that clinicians in low-resource settings also make most decisions based on laboratory testing.[3][4] Despite the importance of laboratory test results in clinical decision-making, little effort has been made in low-resource settings to improve the entire laboratory testing process, which starts when the test is first ordered and ends when the results are interpreted and a clinical decision is made.[5]\nLaboratory errors include a wide variety of mistakes in the testing process and have no universally accepted definition. We define a laboratory error as any event or mistake that leads to failure to perform a laboratory test, misdiagnosis of a laboratory test, or delayed reporting of laboratory test results. In 2001, it was estimated that laboratory errors accounted for $200 to $400 million in American healthcare expenditures per annum.[6] Since then, the rate of utilization of laboratory services has increased, making the reduction of laboratory errors a significant opportunity for cost reduction and healthcare quality improvement.\nRecent studies have tried to categorize errors using phases of the total testing process, which comprises pre-analytical, analytical, and post-analytical phases.[7] The pre-analytical phase covers all activities from when the test is ordered to when the specimen is delivered to the laboratory for testing. The analytical phase covers the activities involved in the actual testing of the specimen, and the post-analytical phase involves the reporting and interpretation of the laboratory result. Among the phases of the total testing process, it has been observed that most laboratory errors happen outside of the analytical phase.[8] An example of an error outside the analytical phase is the mislabeling of a specimen, which could happen during the drawing of a sample in the pre-analytical phase. While error rates vary between health facilities, it is estimated that 32% to 75% of all laboratory errors happen in the pre-analytical phase.[9] Error rates in the analytical phase are estimated in the range of 13% to 32% and in the post-analytical phase in the range of 9% to 31%.[9]\nInformatics interventions may be useful in reducing such laboratory errors. Examples of such interventions are computer-aided ordering of laboratory tests, barcode labeling of specimen tubes, and automated reporting of laboratory test results. These interventions are often provided using computer systems that allow physicians to order diagnostic tests, medications, and other procedures, commonly referred to as computerized provider order entry (CPOE)[10], which is often a part of a larger electronic health record system. However, such comprehensive electronic health record systems have low penetration in low-resource settings where the burden of disease is high and laboratory errors are further exacerbated by poor infrastructure, shortages in trained workforce, and informational challenges.[1][11]\nAlthough laboratory information systems (LIS) have been shown to help reduce laboratory errors, little information is available on the implementation of these in low-resource settings. Furthermore, most descriptions of LIS implementations in low-resource settings focus on the analytical phase of the total testing process.[12] In this article, we describe preliminary work in developing a LIS that addresses problems using informatics interventions to support all phases of the total testing process in a low-resource setting with no pre-existing CPOE system.\n\nMethods \nEthical considerations \nThe work associate with this article followed all ethical standards for research without direct contact with human or animal subjects.\n\nSetting \nWe implemented a LIS at the Kamuzu Central Hospital (KCH) in Lilongwe, Malawi, between January and March 2015. The Kamuzu Central Hospital is a 750-bed government-operated referral hospital. The laboratory at KCH comprises eight departments: microbiology, parasitology, serology, hematology, molecular biology, blood bank, flow cytometry, and biochemistry. These departments perform laboratory tests for both outpatients and inpatients at the hospital and conducted 242,242 tests between July 1, 2010 and June 30, 2011.[13] The system described in this article was piloted in the outpatient tuberculosis clinic of the hospital and the microbiology department of the laboratory starting on March 31, 2015.\n\nUser requirements and system capabilities \nRequirements for the LIS were provided by laboratory technicians in the form of user stories. A user story is a succinct way of representing a task that a user will want to perform using an information resource.[14] It includes the role of the user, the task or action, and the benefit, goal, or achievement. An example of a user story in this context is:\n\nAs a laboratory technician, I want to know which specimen was drawn first so that I can prioritize it for analysis to reduce the number of non-viable specimens.\nWe compiled a consolidated list of user stories for each phase of the total testing process. We used that list to define a set of functionality requirements from the laboratory technicians.\nTo ensure that no core functionality was omitted from the specifications, we leveraged the Laboratory Information System Functionality Assessment Toolkit (LIS-FAT) developed by the Association of Pathology Informatics. This assessment toolkit provides 850 declarative statements that describe the functions that a LIS should possess.[15] An example of a functionality statement from LIS-FAT is:\n\nA laboratory information system should provide intelligent sample labeling that groups samples based on the test to be done and prints them out.\nThe LIS-FAT was originally intended for use as a LIS evaluation checklist. However, in our implementation, we repurposed it to define capabilities for the proposed system. Furthermore, we recognized that the LIS-FAT was primarily developed for use in a setting with adequate resources, and some aspects of it may not be well suited for a low-resource setting. We therefore assessed the LIS-FAT functionality statements and selected those that focused on direct user needs and were most applicable in a low-resource setting. Special effort was made to ensure that major functional categories of the LIS-FAT were not overlooked. This resulted in a customized LIS-FAT applicable to a low-resource setting, with the declarative statements describing the core requirements for LIS in this setting.\nTo elucidate the dependencies that could drive the design phase, all functionality statements created in this step were grouped into high, medium, and low priority categories by a group of laboratory management personnel. This helped determine the order in which the functionality would be implemented to ensure that the most important functionality was implemented first.\n\nSystem design and development \nLaboratory information system software can be commercial, open-source, or home-grown. We chose to build on existing open-source LIS software and customize it based on our functional requirements. Before any functionality was implemented, we conducted a design validation study of two open-source LIS software systems to determine the extent to which they implemented the required functionality for the KCH laboratory.[16] These systems were Open Enterprise Laboratory Information System (OpenELIS) and Basic Laboratory Information System (C4G BLIS).[12][17] We assessed and ranked the systems based on the number of functionality requirements that they satisfied for the LIS implementation at KCH. A functionality requirement was considered satisfied if the LIS had a feature that could be used to achieve the goal of that requirement. The choice between the systems was based on the total number of required functions that each of the systems possessed. The system with the most functionality requirements was selected as the foundation upon which the LIS implementation at KCH would be built, and a comprehensive design was made around it to ensure that all functional requirements were met.\nWe also realized that information systems frequently emphasize the collection and use of data for reporting purposes and streamlining workflow. However, the use of such systems does not necessarily result in improved outcomes. To maximize the value of the LIS, we considered problems identified in the laboratory testing process and described in previous publications.[11][13] Targeted informatics interventions were developed and incorporated into the system\u2019s design to address each of these problems.\nUpon completion of the system design, a team of three developers (C.K., K.K., T.M.M.) iteratively implemented and integrated the remaining functionality over eight weeks from mid-January to mid-March 2015. During this time, clinicians and laboratory personnel provided initial feedback, which we used to refine the user interfaces for the new features.\n\nResults \nUser requirements and system capabilities \nA list of 34 user stories was compiled and mapped into functionality statements for the KCH LIS implementation. An additional 41 statements were added from our review of the LIS-FAT statements. The selected LIS-FAT statements had direct user benefits in keeping with the user stories provided by laboratory technicians and were most suitable for LIS implementation in low-resource settings. These 75 statements formed the core functionality requirements for the LIS implementation at KCH.\n\nSystem design and development \nIn our design validation study, we independently assessed C4G BLIS and OpenELIS against our set of 75 functionality requirements for the KCH LIS. C4G BLIS met 25 (33%) of the functionality requirements and OpenELIS met 22 (29.3%). A detailed breakdown of the functionality that each system had is presented in Table 1.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 1. Functionality assessment of two open-source laboratory information systems for the Kamuzu Central Hospital laboratory testing process, Malawi, 2015\n\n\n\nFollowing the design validation study, C4G BLIS was selected as the base software for the LIS implementation at KCH. To support the pre-analytical and post-analytical phases, we built clinician-facing laboratory order entry and results reporting software modules. Since functionality was already delineated by phases, each phase could be easily conceptualized as an independent component in a larger system. The decision to adopt a modular approach was further driven by the understanding that modules are easier to maintain than a single monolithic system. With the modular approach, any software module can be easily replaced should a more suitable alternative be identified. For example, if another type of LIS software was chosen to replace the customized C4G BLIS at KCH, it could be easily integrated because of the modular approach. This provides flexibility for future improvements of the system. Each module also addressed specific challenges with targeted informatics interventions. A summary of these is provided in Table 2.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 2. Problems in the laboratory testing process and interventions implemented in the laboratory information system to address them at the Kamuzu Central Hospital, Malawi, 2015\n\n\n\nDeveloping bedside solutions \nTo facilitate the bedside use of the LIS, we designed custom hardware in the form of a mobile workstation that clinicians could use for ordering laboratory tests and reviewing laboratory results. The mobile workstation is equipped with a low-cost nine-inch tablet computer, a barcode scanner, and a thermal label printer, as shown in Figure 1. To provide complete mobility, the workstation is powered by batteries and does not need to be plugged in to a power outlet during use. The batteries are charged between ward rounds when the mobile workstation is not in use. The mobile workstation also provides room for the clinicians and nurses to easily carry around all medical supplies and consumables required for specimen collection during ward rounds. The provision of space for medical supplies was made as a value addition for the medical personnel and eliminated the need for a separate cart for medical supplies.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig 1. A mobile workstation equipped with a tablet computer, label printer and barcode scanner, Kamuzu Central Hospital, Malawi, 2015\n\n\n\nTo provide visibility into the status of laboratory tests and results at each stage of the testing process, we also built a dashboard application. This application runs on Raspberry Pi mini-computers connected to 23-inch screens that are mounted in relevant work areas both in the laboratory as well as in the hospital wards. On the screen, we display context-specific work lists such that each user only sees the processes in which they are involved and on which they must act. For example, the dashboard in the microbiology department only shows specimens that require microbiological tests and not any other specimens. A screenshot of the dashboard is provided in Figure 2. \n\r\n\n\n\n\n\n\n\n\n\n\n Fig 2. An example of a nursing station dashboard at the Kamuzu Central Hospital, Malawi, 2015\n\n\n\nFigure 3 depicts how the dashboard application, C4G BLIS and the laboratory order entry and results reporting modules are integrated.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig 3. Architecture of laboratory information system implementation at the Kamuzu Central Hospital, Malawi, 2015\n\n\n\nLaboratory testing workflow with the laboratory information system \nThe testing process begins with the clinical decision to order a laboratory test to assess the patient\u2019s condition. To initiate an order, the clinician uses a unique national patient identifier in the form of a barcode to open the patient\u2019s record in the order entry module. Patient identifiers are issued to patients on arrival at the hospital after completion of a one-time patient registration process, which generates an adhesive label to be affixed to a patient\u2019s personal health passport. A health passport is a paper-based continuity of care document kept by the patient. The framework for uniquely identifying patients at KCH has been described in detail elsewhere.[18] Scanning the patient\u2019s barcode opens the patient\u2019s summary in the order entry module displaying the patient\u2019s past test orders and their status, including test results when available. From this page, the clinician can place new test orders and initiate the pre-analytical phase of the total testing process.\nIn addition to test ordering, the order entry module also maintains and displays an up-to-date catalog of all the tests that are currently being offered at the facility. This intervention helps prevent ordering of tests that are unavailable in the laboratory. Once the test order has been placed through the order entry module, a Health Level 7 message is sent to the C4G BLIS server via a Health Level 7 message router to record the test order. The C4G BLIS server responds by issuing an accession number for the specimen, which is printed on a label in barcode and human readable form together with other test order details during specimen collection. The label is then manually affixed to the specimen container. This accession number is used to track the specimen throughout the testing process. The time at which the specimen label is printed is used in the system as the approximate time when the specimen was collected.\nOnce the order has been placed, it appears on the relevant nursing station dashboard as well as the laboratory reception dashboard. This was done to provide a visual cue for the laboratory receptionist to anticipate incoming specimens from the ward while at the same time reminding nursing station staff that they need to collect and transport the specimens to the laboratory. Once a specimen has been collected, the dashboard displays its viability based on how long it has been since it was drawn. This serves as a reminder for the nursing staff to bring the specimens to the laboratory on time. The specimen will continue to show on the nursing station dashboard until it has been received at the laboratory reception.\nWhen the specimen arrives at the laboratory, the laboratory receptionist scans the barcode and performs a visual inspection of the specimen container and test order documentation. Based on this, the receptionist determines whether the specimen should be accepted or rejected. For example, a specimen can be rejected if it is no longer viable depending on when it was drawn and the type of test that was ordered. When a specimen is rejected, a notification appears on the nursing station dashboard informing the nursing staff to redraw the specimen. If the specimen is accepted at laboratory reception, it is sent to the appropriate laboratory department for analysis and an entry is added to that department\u2019s dashboard. This is the beginning of the analytical phase. The department dashboard acts as a dynamic worklist informing laboratory technicians of tests that need to be run and results that have yet to be recorded.\nOnce a specimen has been analyzed, the laboratory technician enters the result using a touchscreen workstation in the laboratory department to complete the analytical phase of the testing process. The nursing staff are notified of the new result through the nursing station dashboard and can now print out the test result and affix it to the patient\u2019s health passport or medical chart for review by the clinician.\n\nDiscussion \nIn this article, we have described the process used to implement a LIS in a low-resource setting, specifically at KCH in Lilongwe, Malawi. We aimed to demonstrate a problem-driven approach that implements individual informatics interventions to support the laboratory testing process in a low-resource setting. We demonstrated this by piloting a system that supported the entire testing process for outpatient tuberculosis screening tests at KCH.\nAn example of a problem that was addressed in this implementation is the incorrect use of specimen containers for various tests. To address this problem, a picture of the correct container and required specimen volume for each test type was presented to the users during specimen drawing as an electronic job aid. These two interventions addressed the cause of 84% of all untestable specimens at KCH that we reported in a previous article.[13]\nWhile the pilot implementation in the outpatient tuberculosis clinic achieved our main goal, it also limited our ability to measure the impact of the interventions. For instance, sputum is the only specimen type collected in the outpatient tuberculosis clinic at KCH. Therefore, we could not measure the impact of the specimen container decision support on specimen viability. Furthermore, since this ward serves outpatients, the laboratory turnaround time for these tests is not an accurate measure of process efficiency as the review of the results depends on when the patient returns to the hospital, and not when the actual test result itself becomes available. Despite these limitations, there are several points worth noting from this work.\nA set of 75 functionality requirements (Online Appendix 1) for LIS in low-resource settings was produced as part of this project. This contains significantly fewer requirements than the 850 statements found in the LIS-FAT document, which are more appropriate for high-resource settings. These 75 requirements can be used by other implementers in low-resource settings to design and evaluate their own LIS.\nA further point can be made about the actual design of the implementation. The use of distinct modules separated by the phases of the total testing process offers several benefits. Not only can the modules be more easily maintained, they can also be easily replaced should better alternatives be identified. This offers significant benefits going forward as the implementation is not tightly coupled to any single piece of software.\nThis implementation further highlights the benefits that open-source software provides with regard to systems implementation in low-resource settings. Software development takes time and is expensive. However, using existing open-source software has the potential to vastly reduce both effort and cost. For instance, design and development took only 10 weeks because existing software was reused for some parts of our system. This could have been significantly longer if everything was built from scratch. Therefore, we recommend that other implementers in low-resource settings find ways of making use of the many open-source products in the health informatics community as this can reduce their effort and expenditure.\nUsing cheaper alternatives is a common approach to cost reduction. In this implementation, we did this by using tablets that cost $60.00 for the workstations instead of the $650.00 touchscreen clinical workstations that we have used in the past. The tablets presented a significant price reduction and other desirable qualities like the ability to easily be mounted on the mobile workstation. There seemed to be no significant problems with the tablets during the testing phase. However, when we deployed them in the hospital, the tablets often stopped responding and would occasionally power down during use without warning. This led to the loss of all current work that clinical staff had done related to the current patient or specimen and was very disruptive to the workflow.\nOur experience with the tablets emphasizes the need for rigor in the testing of new hardware before deployment. In the next iteration, we will address this by replacing the tablets with Raspberry Pi mini-computers and 10.1-inch touchscreen displays. We have comprehensively tested this solution and believe that these new computers will perform more reliably than the tablets and will not significantly inflate the expenditure on the project as they cost less than $200.00 each.[19]\nThe main limitation of this implementation was our inability to measure the impact that the interventions had on various laboratory key performance indicators such as turnaround times for laboratory tests. In addition, we did not deploy the mobile workstation in the inpatient wards for use during ward rounds. This was mainly due to dependencies that had to be met before deploying the system to inpatient wards at KCH. In the future, we intend to perform field usability evaluations for the mobile workstations and problem impact studies to quantify the effect of the various interventions on laboratory key performance indicators.\nLessons learned from this pilot have informed the continuing scale-up of LIS implementations in Malawi. Those lessons learned include:\n\n An electronic laboratory information system that supports the entire testing process can work in the absence of an electronic medical records system.\n Leveraging individual interventions to solve systemic process challenges can provide benefits to clinical staff that incentivize them to continue using the interventions after implementation.\n Pre-existing functionality assessment toolkits can provide a foundation for building better tools to improve workflows and processes.\nA revised version of this system has now been deployed in three central hospitals and four district hospitals.[20] Revisions have focused on improving operations in the analytical phase by interfacing instruments to the LIS. Future efforts will focus on maximizing the benefits in the pre-analytical and post-analytical phases where most laboratory errors occur.\n\nAcknowledgements \nThe authors would like to thank the clinical and laboratory staff at Kamuzu Central Hospital (KCH), and the Laboratory Information Management Systems Program Secretariat.\n\nCompeting interests \nThe authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.\n\nAuthors\u2019 contributions \nT.M.M. drafted the manuscript and assisted in the design and development of all the required software components. F.E.C. coordinated the implementation in the Kamuzu Central Hospital (KCH) laboratory and influenced the design through original contributions and critical feedback. J.U.E. and S.L.M. analysed open-source laboratory information system software for potential use at KCH. C.K. and K.K. assisted in the design and development of all the required software components. T.E.M. coordinated project activities and oversaw implementation. G.P.D. conceived the original idea and designed custom hardware for the implementation. All authors critically revised the manuscript.\n\nSources of support \nThis project was supported by the President\u2019s Emergency Plan for AIDS Relief through the Centers for Disease Control and Prevention under the terms of 3U2GGH000729-02S1.\nThe authors acknowledge the United States Centers for Disease Control and Prevention which funded the development of the LIS at KCH through their cooperative agreements with the Malawi Ministry of Health and Baobab Health Trust.\n\nData availability statement \nAll software described in this article has been made available on GitHub. The list of functionality statements for laboratory information systems in low-resource settings has been included as supplemental material (Online Appendix 1). No further data were created or analysed in this study.\n\nDisclaimer \nThe findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the funding agencies.\n\nReferences \n\n\n\u2191 1.0 1.1 Becich, M.J. (2000). \"Information management: moving from test results to clinical information\". Clinical Leadership & Management Review 14 (6): 296\u2013300. PMID 11210218.   \n\n\u2191 Hallworth, M.J. (2011). \"The '70% claim': what is the evidence base?\". Annals of Clinical Biochemistry 48 (Pt. 6): 487\u20138. doi:10.1258\/acb.2011.011177. PMID 22045648.   \n\n\u2191 Wilson, M.L.; Fleming, K.A.; Kuti, M.A. et al. (2018). \"Access to pathology and laboratory medicine services: A crucial gap\". Lancet 391 (10133): 1927\u201338. doi:10.1016\/S0140-6736(18)30458-6. PMID 29550029.   \n\n\u2191 Moyo, K.;, Porter, C.; Chilima, B. et al. (2015). \"Use of laboratory test results in patient management by clinicians in Malawi\". African Journal of Laboratory Medicine 4 (1): 277. doi:10.4102\/ajlm.v4i1.277. PMC PMC4870597. PMID 27213139. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4870597 .   \n\n\u2191 Price, C.P.; John, A.S.; Christenson, R. et al. (2016). \"Leveraging the real value of laboratory medicine with the value proposition\". Clinical Chimica Acta 462: 183\u20136. doi:10.1016\/j.cca.2016.09.006. PMID 27649855.   \n\n\u2191 Bologna, L.; Hardy, G.; Mutter, M. (2001). \"Reducing specimen and medication error with handheld technology\". Proceedings of the 2001 HIMSS Annual Conference.   \n\n\u2191 Plebani, M. (2009). \"Exploring the iceberg of errors in laboratory medicine\". Clinical Chimica Acta 404 (1): 16\u201323. doi:10.1016\/j.cca.2009.03.022. PMID 19302995.   \n\n\u2191 Plebani, M. (2006). \"Errors in clinical laboratories or errors in laboratory medicine?\". Clinical Chemistry and Laboratory Medicine 44 (6): 750\u20139. doi:10.1515\/CCLM.2006.123. PMID 16729864.   \n\n\u2191 9.0 9.1 The Lewin Group (May 2008). \"Laboratory Medicine: A National Status Report\" (PDF). Centers for Disease Control and Prevention. https:\/\/www.cdc.gov\/labbestpractices\/pdfs\/2007-status-report-laboratory_medicine_-_a_national_status_report_from_the_lewin_group_updated_2008-9.pdf .   \n\n\u2191 Baron, J.M.; Dighe, A.S. (2011). \"Computerized provider order entry in the clinical laboratory\". Journal of Pathology Informatics 2: 35. doi:10.4103\/2153-3539.83740. PMC PMC3162747. PMID 21886891. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162747 .   \n\n\u2191 11.0 11.1 Petrose, L.G.; Fisher, A.M.; Douglas, G.P. et al. (2016). \"Assessing Perceived Challenges to Laboratory Testing at a Malawian Referral Hospital\". American Journal of Tropical Medicine and Hygiene 94 (6): 1426-32. doi:10.4269\/ajtmh.15-0867. PMC PMC3162747. PMID 27022150. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162747 .   \n\n\u2191 12.0 12.1 Vempala, S.; Chopra, N.; Rajagopal, A. et al. (2016). \"C4G BLIS: Health Care Delivery via Iterative Collaborative Design in Resource-constrained Settings\". Proceedings of the Eighth International Conference on Information and Communication Technologies and Development: 21. doi:10.1145\/2909609.2909657.   \n\n\u2191 13.0 13.1 13.2 Driessen, J.; Limula, H.; Gadabu, O.J. et al. (2015). \"Informatics solutions for bridging the gap between clinical and laboratory services in a low-resource setting\". African Journal of Laboratory Medicine 4 (1): 176. doi:10.4102\/ajlm.v4i1.176.   \n\n\u2191 Cohn, M. (2004). User Stories Applied: For Agile Software Development (1st ed.). Addison-Wesley Professional. ISBN 9780321205681.   \n\n\u2191 Tuthill, J.M.; Friedman, B.A.; Balis, U.J. et al. (2014). \"The laboratory information system functionality assessment tool: Ensuring optimal software support for your laboratory\". Journal of Pathology Informatics 5 (1): 7. doi:10.4103\/2153-3539.127819. PMC PMC3986538. PMID 24741466. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3986538 .   \n\n\u2191 Friedman, C.P.; Wyatt, J., ed. (2006). Evaluation Methods in Biomedical Informatics. Springer-Verlag. doi:10.1007\/0-387-30677-3. ISBN 9780387306773.   \n\n\u2191 Monu, R. (27 May 2010). \"Design and implementation of a basic laboratory information system for resource-limited settings\". SMARTech: Georgia Tech Theses and Dissertations. http:\/\/hdl.handle.net\/1853\/34792 .   \n\n\u2191 Douglas, G.P.; Gadabu, O.J.; Joukes, S. et al. (2010). \"Using touchscreen electronic medical record systems to support and monitor national scale-up of antiretroviral therapy in Malawi\". PLoS Medicine 7 (8): e1000319. doi:10.1371\/journal.pmed.1000319. PMC PMC2919419. PMID 20711476. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2919419 .   \n\n\u2191 Mtonga, T.; Abaye, M.; Rosko, S.C.; Douglas, G.P. (2018). \"A comparative usability study of two touchscreen clinical workstations for use in low resource settings\". Journal of Health Informatics in Africa 5 (2): 78\u201384. doi:10.12856\/JHIA-2018-v5-i2-209.   \n\n\u2191 Baobab Health. \"Baobab Health Electronic Medical Records Systems Foot Print\". http:\/\/www.baobabhealth.org\/#work .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The article mentions \"Online Appendix 1,\" but it could not be found on the journal website. Neither is it clear which GitHub repository applies to their developed software, which the authors claim is on GitHub.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\">https:\/\/www.limswiki.org\/index.php\/Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on information technology\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 1 June 2020, at 20:36.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 511 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","eb76e424fc635068c27be185a754eac7_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Design and implementation of a clinical laboratory information system in a low-resource setting<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: Reducing <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> errors presents a significant opportunity for both cost reduction and healthcare <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_(business)\" title=\"Quality (business)\" class=\"wiki-link\" data-key=\"c4ac43430d1c3a3a15d1255257aaea37\">quality<\/a> improvement. This is particularly true in low-resource settings where laboratory errors are further exacerbated by poor infrastructure and shortages in a trained workforce. <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics_(academic_field)\" title=\"Informatics (academic field)\" class=\"wiki-link\" data-key=\"0391318826a5d9f9a1a1bcc88394739f\">informatics<\/a> interventions can be used to address some of the sources of laboratory errors.\n<\/p><p><b>Objectives<\/b>: This article describes the development process for a clinical <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS) that leverages informatics interventions to address problems in the laboratory testing process at a <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> in a low-resource setting.\n<\/p><p><b>Methods<\/b>: We designed interventions using informatics methods for previously identified problems in the laboratory testing process at a <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_laboratory\" title=\"Clinical laboratory\" class=\"wiki-link\" data-key=\"307bcdf1bdbcd1bb167cee435b7a5463\">clinical laboratory<\/a> in a low-resource setting. First, we reviewed a pre-existing LIS functionality assessment toolkit and consulted with laboratory personnel. This provided requirements that were developed into a LIS with interventions designed to address the problems that had been identified. We piloted the LIS at the Kamuzu Central Hospital in Lilongwe, Malawi.\n<\/p><p><b>Results<\/b>: We implemented a series of informatics interventions in the form of a LIS to address sources of laboratory errors and support the entire laboratory testing process. Custom hardware was built to support the ordering of laboratory tests and review of laboratory test results.\n<\/p><p><b>Conclusion<\/b>: Our experience highlights the potential of using informatics interventions to address systemic problems in the laboratory testing process in low-resource settings. Implementing these interventions may require innovation of new hardware to address various contextual issues. We strongly encourage thorough testing of such innovations to reduce the risk of failure when implemented.\n<\/p><p><b>Keywords<\/b>: low-resource setting, laboratory testing, laboratory information system, Malawi, informatics interventions\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">Laboratory<\/a> testing plays a vital role in clinical decision-making. It is estimated that up to 70% of medical decisions in high-resource healthcare settings are made based on <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_laboratory\" title=\"Clinical laboratory\" class=\"wiki-link\" data-key=\"307bcdf1bdbcd1bb167cee435b7a5463\">clinical laboratory<\/a> test results.<sup id=\"rdp-ebb-cite_ref-BecichInform00_1-0\" class=\"reference\"><a href=\"#cite_note-BecichInform00-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HallworthThe70_11_2-0\" class=\"reference\"><a href=\"#cite_note-HallworthThe70_11-2\">[2]<\/a><\/sup> Even though access to clinical laboratory services is comparatively lower in low-resource settings, studies show that clinicians in low-resource settings also make most decisions based on laboratory testing.<sup id=\"rdp-ebb-cite_ref-WilsonAccess18_3-0\" class=\"reference\"><a href=\"#cite_note-WilsonAccess18-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MoyoUse15_4-0\" class=\"reference\"><a href=\"#cite_note-MoyoUse15-4\">[4]<\/a><\/sup> Despite the importance of laboratory test results in clinical decision-making, little effort has been made in low-resource settings to improve the entire laboratory testing process, which starts when the test is first ordered and ends when the results are interpreted and a clinical decision is made.<sup id=\"rdp-ebb-cite_ref-PriceLever16_5-0\" class=\"reference\"><a href=\"#cite_note-PriceLever16-5\">[5]<\/a><\/sup>\n<\/p><p>Laboratory errors include a wide variety of mistakes in the testing process and have no universally accepted definition. We define a laboratory error as any event or mistake that leads to failure to perform a laboratory test, misdiagnosis of a laboratory test, or delayed reporting of laboratory test results. In 2001, it was estimated that laboratory errors accounted for $200 to $400 million in American healthcare expenditures per annum.<sup id=\"rdp-ebb-cite_ref-BolognaReduc01_6-0\" class=\"reference\"><a href=\"#cite_note-BolognaReduc01-6\">[6]<\/a><\/sup> Since then, the rate of utilization of laboratory services has increased, making the reduction of laboratory errors a significant opportunity for cost reduction and healthcare <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_(business)\" title=\"Quality (business)\" class=\"wiki-link\" data-key=\"c4ac43430d1c3a3a15d1255257aaea37\">quality<\/a> improvement.\n<\/p><p>Recent studies have tried to categorize errors using phases of the total testing process, which comprises pre-analytical, analytical, and post-analytical phases.<sup id=\"rdp-ebb-cite_ref-PlebaniExplor09_7-0\" class=\"reference\"><a href=\"#cite_note-PlebaniExplor09-7\">[7]<\/a><\/sup> The pre-analytical phase covers all activities from when the test is ordered to when the specimen is delivered to the laboratory for testing. The analytical phase covers the activities involved in the actual testing of the specimen, and the post-analytical phase involves the reporting and interpretation of the laboratory result. Among the phases of the total testing process, it has been observed that most laboratory errors happen outside of the analytical phase.<sup id=\"rdp-ebb-cite_ref-PlebaniErrors06_8-0\" class=\"reference\"><a href=\"#cite_note-PlebaniErrors06-8\">[8]<\/a><\/sup> An example of an error outside the analytical phase is the mislabeling of a specimen, which could happen during the drawing of a sample in the pre-analytical phase. While error rates vary between health facilities, it is estimated that 32% to 75% of all laboratory errors happen in the pre-analytical phase.<sup id=\"rdp-ebb-cite_ref-LewinLab08_9-0\" class=\"reference\"><a href=\"#cite_note-LewinLab08-9\">[9]<\/a><\/sup> Error rates in the analytical phase are estimated in the range of 13% to 32% and in the post-analytical phase in the range of 9% to 31%.<sup id=\"rdp-ebb-cite_ref-LewinLab08_9-1\" class=\"reference\"><a href=\"#cite_note-LewinLab08-9\">[9]<\/a><\/sup>\n<\/p><p>Informatics interventions may be useful in reducing such laboratory errors. Examples of such interventions are computer-aided ordering of laboratory tests, barcode labeling of specimen tubes, and automated reporting of laboratory test results. These interventions are often provided using computer systems that allow physicians to order diagnostic tests, medications, and other procedures, commonly referred to as <a href=\"https:\/\/www.limswiki.org\/index.php\/Computerized_physician_order_entry\" title=\"Computerized physician order entry\" class=\"wiki-link\" data-key=\"f9e67e685f2b29f79e9b0991330f2b10\">computerized provider order entry<\/a> (CPOE)<sup id=\"rdp-ebb-cite_ref-BaronCompu11_10-0\" class=\"reference\"><a href=\"#cite_note-BaronCompu11-10\">[10]<\/a><\/sup>, which is often a part of a larger <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health record<\/a> system. However, such comprehensive electronic health record systems have low penetration in low-resource settings where the burden of disease is high and laboratory errors are further exacerbated by poor infrastructure, shortages in trained workforce, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">informational<\/a> challenges.<sup id=\"rdp-ebb-cite_ref-BecichInform00_1-1\" class=\"reference\"><a href=\"#cite_note-BecichInform00-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PetroseAss16_11-0\" class=\"reference\"><a href=\"#cite_note-PetroseAss16-11\">[11]<\/a><\/sup>\n<\/p><p>Although <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information systems<\/a> (LIS) have been shown to help reduce laboratory errors, little information is available on the implementation of these in low-resource settings. Furthermore, most descriptions of LIS implementations in low-resource settings focus on the analytical phase of the total testing process.<sup id=\"rdp-ebb-cite_ref-VempalaC4G16_12-0\" class=\"reference\"><a href=\"#cite_note-VempalaC4G16-12\">[12]<\/a><\/sup> In this article, we describe preliminary work in developing a LIS that addresses problems using <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics_(academic_field)\" title=\"Informatics (academic field)\" class=\"wiki-link\" data-key=\"0391318826a5d9f9a1a1bcc88394739f\">informatics<\/a> interventions to support all phases of the total testing process in a low-resource setting with no pre-existing CPOE system.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Ethical_considerations\">Ethical considerations<\/span><\/h3>\n<p>The work associate with this article followed all ethical standards for research without direct contact with human or animal subjects.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Setting\">Setting<\/span><\/h3>\n<p>We implemented a LIS at the Kamuzu Central Hospital (KCH) in Lilongwe, Malawi, between January and March 2015. The Kamuzu Central Hospital is a 750-bed government-operated referral hospital. The laboratory at KCH comprises eight departments: microbiology, parasitology, serology, <a href=\"https:\/\/www.limswiki.org\/index.php\/Hematology\" title=\"Hematology\" class=\"wiki-link\" data-key=\"de8b49c7b0be3cec33af362e763b9b0c\">hematology<\/a>, molecular biology, blood bank, flow cytometry, and biochemistry. These departments perform laboratory tests for both outpatients and inpatients at the hospital and conducted 242,242 tests between July 1, 2010 and June 30, 2011.<sup id=\"rdp-ebb-cite_ref-DriessenInform15_13-0\" class=\"reference\"><a href=\"#cite_note-DriessenInform15-13\">[13]<\/a><\/sup> The system described in this article was piloted in the outpatient tuberculosis clinic of the hospital and the microbiology department of the laboratory starting on March 31, 2015.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"User_requirements_and_system_capabilities\">User requirements and system capabilities<\/span><\/h3>\n<p>Requirements for the LIS were provided by laboratory technicians in the form of user stories. A user story is a succinct way of representing a task that a user will want to perform using an information resource.<sup id=\"rdp-ebb-cite_ref-CohnUser05_14-0\" class=\"reference\"><a href=\"#cite_note-CohnUser05-14\">[14]<\/a><\/sup> It includes the role of the user, the task or action, and the benefit, goal, or achievement. An example of a user story in this context is:\n<\/p>\n<dl><dd><i>As a laboratory technician, I want to know which specimen was drawn first so that I can prioritize it for analysis to reduce the number of non-viable specimens.<\/i><\/dd><\/dl>\n<p>We compiled a consolidated list of user stories for each phase of the total testing process. We used that list to define a set of functionality requirements from the laboratory technicians.\n<\/p><p>To ensure that no core functionality was omitted from the specifications, we leveraged the Laboratory Information System Functionality Assessment Toolkit (LIS-FAT) developed by the Association of Pathology Informatics. This assessment toolkit provides 850 declarative statements that describe the functions that a LIS should possess.<sup id=\"rdp-ebb-cite_ref-TuthillTheLab14_15-0\" class=\"reference\"><a href=\"#cite_note-TuthillTheLab14-15\">[15]<\/a><\/sup> An example of a functionality statement from LIS-FAT is:\n<\/p>\n<dl><dd><i>A laboratory information system should provide intelligent sample labeling that groups samples based on the test to be done and prints them out.<\/i><\/dd><\/dl>\n<p>The LIS-FAT was originally intended for use as a LIS evaluation checklist. However, in our implementation, we repurposed it to define capabilities for the proposed system. Furthermore, we recognized that the LIS-FAT was primarily developed for use in a setting with adequate resources, and some aspects of it may not be well suited for a low-resource setting. We therefore assessed the LIS-FAT functionality statements and selected those that focused on direct user needs and were most applicable in a low-resource setting. Special effort was made to ensure that major functional categories of the LIS-FAT were not overlooked. This resulted in a customized LIS-FAT applicable to a low-resource setting, with the declarative statements describing the core requirements for LIS in this setting.\n<\/p><p>To elucidate the dependencies that could drive the design phase, all functionality statements created in this step were grouped into high, medium, and low priority categories by a group of laboratory management personnel. This helped determine the order in which the functionality would be implemented to ensure that the most important functionality was implemented first.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"System_design_and_development\">System design and development<\/span><\/h3>\n<p>Laboratory information system software can be commercial, open-source, or home-grown. We chose to build on existing open-source LIS software and customize it based on our functional requirements. Before any functionality was implemented, we conducted a design validation study of two open-source LIS software systems to determine the extent to which they implemented the required functionality for the KCH laboratory.<sup id=\"rdp-ebb-cite_ref-FriedmanEval06_16-0\" class=\"reference\"><a href=\"#cite_note-FriedmanEval06-16\">[16]<\/a><\/sup> These systems were Open Enterprise Laboratory Information System (<a href=\"https:\/\/www.limswiki.org\/index.php\/OpenELIS\" title=\"OpenELIS\" class=\"wiki-link\" data-key=\"b11b00d1e7409d55b58e9f82204cc2a3\">OpenELIS<\/a>) and Basic Laboratory Information System (<a href=\"https:\/\/www.limswiki.org\/index.php\/C4G_BLIS\" title=\"C4G BLIS\" class=\"wiki-link\" data-key=\"bfa1af20f4a68dc928bd873f549e440e\">C4G BLIS<\/a>).<sup id=\"rdp-ebb-cite_ref-VempalaC4G16_12-1\" class=\"reference\"><a href=\"#cite_note-VempalaC4G16-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MonuDesign10_17-0\" class=\"reference\"><a href=\"#cite_note-MonuDesign10-17\">[17]<\/a><\/sup> We assessed and ranked the systems based on the number of functionality requirements that they satisfied for the LIS implementation at KCH. A functionality requirement was considered satisfied if the LIS had a feature that could be used to achieve the goal of that requirement. The choice between the systems was based on the total number of required functions that each of the systems possessed. The system with the most functionality requirements was selected as the foundation upon which the LIS implementation at KCH would be built, and a comprehensive design was made around it to ensure that all functional requirements were met.\n<\/p><p>We also realized that information systems frequently emphasize the collection and use of data for reporting purposes and streamlining <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a>. However, the use of such systems does not necessarily result in improved outcomes. To maximize the value of the LIS, we considered problems identified in the laboratory testing process and described in previous publications.<sup id=\"rdp-ebb-cite_ref-PetroseAss16_11-1\" class=\"reference\"><a href=\"#cite_note-PetroseAss16-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DriessenInform15_13-1\" class=\"reference\"><a href=\"#cite_note-DriessenInform15-13\">[13]<\/a><\/sup> Targeted informatics interventions were developed and incorporated into the system\u2019s design to address each of these problems.\n<\/p><p>Upon completion of the system design, a team of three developers (C.K., K.K., T.M.M.) iteratively implemented and integrated the remaining functionality over eight weeks from mid-January to mid-March 2015. During this time, clinicians and laboratory personnel provided initial feedback, which we used to refine the user interfaces for the new features.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"User_requirements_and_system_capabilities_2\">User requirements and system capabilities<\/span><\/h3>\n<p>A list of 34 user stories was compiled and mapped into functionality statements for the KCH LIS implementation. An additional 41 statements were added from our review of the LIS-FAT statements. The selected LIS-FAT statements had direct user benefits in keeping with the user stories provided by laboratory technicians and were most suitable for LIS implementation in low-resource settings. These 75 statements formed the core functionality requirements for the LIS implementation at KCH.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"System_design_and_development_2\">System design and development<\/span><\/h3>\n<p>In our design validation study, we independently assessed C4G BLIS and OpenELIS against our set of 75 functionality requirements for the KCH LIS. C4G BLIS met 25 (33%) of the functionality requirements and OpenELIS met 22 (29.3%). A detailed breakdown of the functionality that each system had is presented in Table 1.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Mtonga_AfricanJofLabMed2019_8-1.jpg\" class=\"image wiki-link\" data-key=\"5a912314ef0d3c5f3b9a96f95544edca\"><img alt=\"Tab1 Mtonga AfricanJofLabMed2019 8-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3f\/Tab1_Mtonga_AfricanJofLabMed2019_8-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 1.<\/b> Functionality assessment of two open-source laboratory information systems for the Kamuzu Central Hospital laboratory testing process, Malawi, 2015<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Following the design validation study, C4G BLIS was selected as the base software for the LIS implementation at KCH. To support the pre-analytical and post-analytical phases, we built clinician-facing laboratory order entry and results reporting software modules. Since functionality was already delineated by phases, each phase could be easily conceptualized as an independent component in a larger system. The decision to adopt a modular approach was further driven by the understanding that modules are easier to maintain than a single monolithic system. With the modular approach, any software module can be easily replaced should a more suitable alternative be identified. For example, if another type of LIS software was chosen to replace the customized C4G BLIS at KCH, it could be easily integrated because of the modular approach. This provides flexibility for future improvements of the system. Each module also addressed specific challenges with targeted informatics interventions. A summary of these is provided in Table 2.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab2_Mtonga_AfricanJofLabMed2019_8-1.jpg\" class=\"image wiki-link\" data-key=\"fa0fcf174d56ac538a698a3d2e616faf\"><img alt=\"Tab2 Mtonga AfricanJofLabMed2019 8-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a5\/Tab2_Mtonga_AfricanJofLabMed2019_8-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 2.<\/b> Problems in the laboratory testing process and interventions implemented in the laboratory information system to address them at the Kamuzu Central Hospital, Malawi, 2015<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Developing_bedside_solutions\">Developing bedside solutions<\/span><\/h3>\n<p>To facilitate the bedside use of the LIS, we designed custom hardware in the form of a mobile workstation that clinicians could use for ordering laboratory tests and reviewing laboratory results. The mobile workstation is equipped with a low-cost nine-inch tablet computer, a barcode scanner, and a thermal label printer, as shown in Figure 1. To provide complete mobility, the workstation is powered by batteries and does not need to be plugged in to a power outlet during use. The batteries are charged between ward rounds when the mobile workstation is not in use. The mobile workstation also provides room for the clinicians and nurses to easily carry around all medical supplies and consumables required for specimen collection during ward rounds. The provision of space for medical supplies was made as a value addition for the medical personnel and eliminated the need for a separate cart for medical supplies.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Mtonga_AfricanJofLabMed2019_8-1.jpg\" class=\"image wiki-link\" data-key=\"3ce31dd41a383eee7a196dd2a50afebc\"><img alt=\"Fig1 Mtonga AfricanJofLabMed2019 8-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/73\/Fig1_Mtonga_AfricanJofLabMed2019_8-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig 1.<\/b> A mobile workstation equipped with a tablet computer, label printer and barcode scanner, Kamuzu Central Hospital, Malawi, 2015<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>To provide visibility into the status of laboratory tests and results at each stage of the testing process, we also built a dashboard application. This application runs on Raspberry Pi mini-computers connected to 23-inch screens that are mounted in relevant work areas both in the laboratory as well as in the hospital wards. On the screen, we display context-specific work lists such that each user only sees the processes in which they are involved and on which they must act. For example, the dashboard in the microbiology department only shows specimens that require microbiological tests and not any other specimens. A screenshot of the dashboard is provided in Figure 2. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Mtonga_AfricanJofLabMed2019_8-1.jpg\" class=\"image wiki-link\" data-key=\"f057c20ae85cee31bdaa25fb7e7eaaa8\"><img alt=\"Fig2 Mtonga AfricanJofLabMed2019 8-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/4c\/Fig2_Mtonga_AfricanJofLabMed2019_8-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig 2.<\/b> An example of a nursing station dashboard at the Kamuzu Central Hospital, Malawi, 2015<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Figure 3 depicts how the dashboard application, C4G BLIS and the laboratory order entry and results reporting modules are integrated.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Mtonga_AfricanJofLabMed2019_8-1.jpg\" class=\"image wiki-link\" data-key=\"ab9247f6f9a3b9aadece26ba35ef23b5\"><img alt=\"Fig3 Mtonga AfricanJofLabMed2019 8-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/08\/Fig3_Mtonga_AfricanJofLabMed2019_8-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig 3.<\/b> Architecture of laboratory information system implementation at the Kamuzu Central Hospital, Malawi, 2015<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Laboratory_testing_workflow_with_the_laboratory_information_system\">Laboratory testing workflow with the laboratory information system<\/span><\/h3>\n<p>The testing process begins with the clinical decision to order a laboratory test to assess the patient\u2019s condition. To initiate an order, the clinician uses a unique national patient identifier in the form of a barcode to open the patient\u2019s record in the order entry module. Patient identifiers are issued to patients on arrival at the hospital after completion of a one-time patient registration process, which generates an adhesive label to be affixed to a patient\u2019s personal health passport. A health passport is a paper-based continuity of care document kept by the patient. The framework for uniquely identifying patients at KCH has been described in detail elsewhere.<sup id=\"rdp-ebb-cite_ref-DouglasUsing10_18-0\" class=\"reference\"><a href=\"#cite_note-DouglasUsing10-18\">[18]<\/a><\/sup> Scanning the patient\u2019s barcode opens the patient\u2019s summary in the order entry module displaying the patient\u2019s past test orders and their status, including test results when available. From this page, the clinician can place new test orders and initiate the pre-analytical phase of the total testing process.\n<\/p><p>In addition to test ordering, the order entry module also maintains and displays an up-to-date catalog of all the tests that are currently being offered at the facility. This intervention helps prevent ordering of tests that are unavailable in the laboratory. Once the test order has been placed through the order entry module, a <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"e0bf845fb58d2bae05a846b47629e86f\">Health Level 7<\/a> message is sent to the C4G BLIS server via a Health Level 7 message router to record the test order. The C4G BLIS server responds by issuing an accession number for the specimen, which is printed on a label in barcode and human readable form together with other test order details during specimen collection. The label is then manually affixed to the specimen container. This accession number is used to track the specimen throughout the testing process. The time at which the specimen label is printed is used in the system as the approximate time when the specimen was collected.\n<\/p><p>Once the order has been placed, it appears on the relevant nursing station dashboard as well as the laboratory reception dashboard. This was done to provide a visual cue for the laboratory receptionist to anticipate incoming specimens from the ward while at the same time reminding nursing station staff that they need to collect and transport the specimens to the laboratory. Once a specimen has been collected, the dashboard displays its viability based on how long it has been since it was drawn. This serves as a reminder for the nursing staff to bring the specimens to the laboratory on time. The specimen will continue to show on the nursing station dashboard until it has been received at the laboratory reception.\n<\/p><p>When the specimen arrives at the laboratory, the laboratory receptionist scans the barcode and performs a visual inspection of the specimen container and test order documentation. Based on this, the receptionist determines whether the specimen should be accepted or rejected. For example, a specimen can be rejected if it is no longer viable depending on when it was drawn and the type of test that was ordered. When a specimen is rejected, a notification appears on the nursing station dashboard informing the nursing staff to redraw the specimen. If the specimen is accepted at laboratory reception, it is sent to the appropriate laboratory department for analysis and an entry is added to that department\u2019s dashboard. This is the beginning of the analytical phase. The department dashboard acts as a dynamic worklist informing laboratory technicians of tests that need to be run and results that have yet to be recorded.\n<\/p><p>Once a specimen has been analyzed, the laboratory technician enters the result using a touchscreen workstation in the laboratory department to complete the analytical phase of the testing process. The nursing staff are notified of the new result through the nursing station dashboard and can now print out the test result and affix it to the patient\u2019s health passport or medical chart for review by the clinician.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>In this article, we have described the process used to implement a LIS in a low-resource setting, specifically at KCH in Lilongwe, Malawi. We aimed to demonstrate a problem-driven approach that implements individual informatics interventions to support the laboratory testing process in a low-resource setting. We demonstrated this by piloting a system that supported the entire testing process for outpatient tuberculosis screening tests at KCH.\n<\/p><p>An example of a problem that was addressed in this implementation is the incorrect use of specimen containers for various tests. To address this problem, a picture of the correct container and required specimen volume for each test type was presented to the users during specimen drawing as an electronic job aid. These two interventions addressed the cause of 84% of all untestable specimens at KCH that we reported in a previous article.<sup id=\"rdp-ebb-cite_ref-DriessenInform15_13-2\" class=\"reference\"><a href=\"#cite_note-DriessenInform15-13\">[13]<\/a><\/sup>\n<\/p><p>While the pilot implementation in the outpatient tuberculosis clinic achieved our main goal, it also limited our ability to measure the impact of the interventions. For instance, sputum is the only specimen type collected in the outpatient tuberculosis clinic at KCH. Therefore, we could not measure the impact of the specimen container decision support on specimen viability. Furthermore, since this ward serves outpatients, the laboratory turnaround time for these tests is not an accurate measure of process efficiency as the review of the results depends on when the patient returns to the hospital, and not when the actual test result itself becomes available. Despite these limitations, there are several points worth noting from this work.\n<\/p><p>A set of 75 functionality requirements (Online Appendix 1) for LIS in low-resource settings was produced as part of this project. This contains significantly fewer requirements than the 850 statements found in the LIS-FAT document, which are more appropriate for high-resource settings. These 75 requirements can be used by other implementers in low-resource settings to design and evaluate their own LIS.\n<\/p><p>A further point can be made about the actual design of the implementation. The use of distinct modules separated by the phases of the total testing process offers several benefits. Not only can the modules be more easily maintained, they can also be easily replaced should better alternatives be identified. This offers significant benefits going forward as the implementation is not tightly coupled to any single piece of software.\n<\/p><p>This implementation further highlights the benefits that open-source software provides with regard to systems implementation in low-resource settings. Software development takes time and is expensive. However, using existing open-source software has the potential to vastly reduce both effort and cost. For instance, design and development took only 10 weeks because existing software was reused for some parts of our system. This could have been significantly longer if everything was built from scratch. Therefore, we recommend that other implementers in low-resource settings find ways of making use of the many open-source products in the health informatics community as this can reduce their effort and expenditure.\n<\/p><p>Using cheaper alternatives is a common approach to cost reduction. In this implementation, we did this by using tablets that cost $60.00 for the workstations instead of the $650.00 touchscreen clinical workstations that we have used in the past. The tablets presented a significant price reduction and other desirable qualities like the ability to easily be mounted on the mobile workstation. There seemed to be no significant problems with the tablets during the testing phase. However, when we deployed them in the hospital, the tablets often stopped responding and would occasionally power down during use without warning. This led to the loss of all current work that clinical staff had done related to the current patient or specimen and was very disruptive to the workflow.\n<\/p><p>Our experience with the tablets emphasizes the need for rigor in the testing of new hardware before deployment. In the next iteration, we will address this by replacing the tablets with Raspberry Pi mini-computers and 10.1-inch touchscreen displays. We have comprehensively tested this solution and believe that these new computers will perform more reliably than the tablets and will not significantly inflate the expenditure on the project as they cost less than $200.00 each.<sup id=\"rdp-ebb-cite_ref-MtongaAComp18_19-0\" class=\"reference\"><a href=\"#cite_note-MtongaAComp18-19\">[19]<\/a><\/sup>\n<\/p><p>The main limitation of this implementation was our inability to measure the impact that the interventions had on various laboratory key performance indicators such as turnaround times for laboratory tests. In addition, we did not deploy the mobile workstation in the inpatient wards for use during ward rounds. This was mainly due to dependencies that had to be met before deploying the system to inpatient wards at KCH. In the future, we intend to perform field usability evaluations for the mobile workstations and problem impact studies to quantify the effect of the various interventions on laboratory key performance indicators.\n<\/p><p>Lessons learned from this pilot have informed the continuing scale-up of LIS implementations in Malawi. Those lessons learned include:\n<\/p>\n<ul><li> An electronic laboratory information system that supports the entire testing process can work in the absence of an electronic medical records system.<\/li>\n<li> Leveraging individual interventions to solve systemic process challenges can provide benefits to clinical staff that incentivize them to continue using the interventions after implementation.<\/li>\n<li> Pre-existing functionality assessment toolkits can provide a foundation for building better tools to improve workflows and processes.<\/li><\/ul>\n<p>A revised version of this system has now been deployed in three central hospitals and four district hospitals.<sup id=\"rdp-ebb-cite_ref-BHBaobabHealth19_20-0\" class=\"reference\"><a href=\"#cite_note-BHBaobabHealth19-20\">[20]<\/a><\/sup> Revisions have focused on improving operations in the analytical phase by interfacing instruments to the LIS. Future efforts will focus on maximizing the benefits in the pre-analytical and post-analytical phases where most laboratory errors occur.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors would like to thank the clinical and laboratory staff at Kamuzu Central Hospital (KCH), and the Laboratory Information Management Systems Program Secretariat.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h3>\n<p>T.M.M. drafted the manuscript and assisted in the design and development of all the required software components. F.E.C. coordinated the implementation in the Kamuzu Central Hospital (KCH) laboratory and influenced the design through original contributions and critical feedback. J.U.E. and S.L.M. analysed open-source laboratory information system software for potential use at KCH. C.K. and K.K. assisted in the design and development of all the required software components. T.E.M. coordinated project activities and oversaw implementation. G.P.D. conceived the original idea and designed custom hardware for the implementation. All authors critically revised the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sources_of_support\">Sources of support<\/span><\/h3>\n<p>This project was supported by the President\u2019s Emergency Plan for AIDS Relief through the Centers for Disease Control and Prevention under the terms of 3U2GGH000729-02S1.\n<\/p><p>The authors acknowledge the United States Centers for Disease Control and Prevention which funded the development of the LIS at KCH through their cooperative agreements with the Malawi Ministry of Health and Baobab Health Trust.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_availability_statement\">Data availability statement<\/span><\/h3>\n<p>All software described in this article has been made available on GitHub. The list of functionality statements for laboratory information systems in low-resource settings has been included as supplemental material (Online Appendix 1). No further data were created or analysed in this study.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Disclaimer\">Disclaimer<\/span><\/h3>\n<p>The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the funding agencies.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-BecichInform00-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BecichInform00_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-BecichInform00_1-1\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Becich, M.J. (2000). \"Information management: moving from test results to clinical information\". <i>Clinical Leadership & Management Review<\/i> <b>14<\/b> (6): 296\u2013300. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11210218\" target=\"_blank\">11210218<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+management%3A+moving+from+test+results+to+clinical+information&rft.jtitle=Clinical+Leadership+%26+Management+Review&rft.aulast=Becich%2C+M.J.&rft.au=Becich%2C+M.J.&rft.date=2000&rft.volume=14&rft.issue=6&rft.pages=296%E2%80%93300&rft_id=info:pmid\/11210218&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HallworthThe70_11-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HallworthThe70_11_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hallworth, M.J. (2011). \"The '70% claim': what is the evidence base?\". <i>Annals of Clinical Biochemistry<\/i> <b>48<\/b> (Pt. 6): 487\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1258%2Facb.2011.011177\" target=\"_blank\">10.1258\/acb.2011.011177<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22045648\" target=\"_blank\">22045648<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+%2770%25+claim%27%3A+what+is+the+evidence+base%3F&rft.jtitle=Annals+of+Clinical+Biochemistry&rft.aulast=Hallworth%2C+M.J.&rft.au=Hallworth%2C+M.J.&rft.date=2011&rft.volume=48&rft.issue=Pt.+6&rft.pages=487%E2%80%938&rft_id=info:doi\/10.1258%2Facb.2011.011177&rft_id=info:pmid\/22045648&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilsonAccess18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilsonAccess18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wilson, M.L.; Fleming, K.A.; Kuti, M.A. et al. (2018). \"Access to pathology and laboratory medicine services: A crucial gap\". <i>Lancet<\/i> <b>391<\/b> (10133): 1927\u201338. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0140-6736%2818%2930458-6\" target=\"_blank\">10.1016\/S0140-6736(18)30458-6<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29550029\" target=\"_blank\">29550029<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Access+to+pathology+and+laboratory+medicine+services%3A+A+crucial+gap&rft.jtitle=Lancet&rft.aulast=Wilson%2C+M.L.%3B+Fleming%2C+K.A.%3B+Kuti%2C+M.A.+et+al.&rft.au=Wilson%2C+M.L.%3B+Fleming%2C+K.A.%3B+Kuti%2C+M.A.+et+al.&rft.date=2018&rft.volume=391&rft.issue=10133&rft.pages=1927%E2%80%9338&rft_id=info:doi\/10.1016%2FS0140-6736%2818%2930458-6&rft_id=info:pmid\/29550029&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoyoUse15-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoyoUse15_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moyo, K.;, Porter, C.; Chilima, B. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4870597\" target=\"_blank\">\"Use of laboratory test results in patient management by clinicians in Malawi\"<\/a>. <i>African Journal of Laboratory Medicine<\/i> <b>4<\/b> (1): 277. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v4i1.277\" target=\"_blank\">10.4102\/ajlm.v4i1.277<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4870597\/\" target=\"_blank\">PMC4870597<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27213139\" target=\"_blank\">27213139<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4870597\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4870597<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Use+of+laboratory+test+results+in+patient+management+by+clinicians+in+Malawi&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Moyo%2C+K.%3B%2C+Porter%2C+C.%3B+Chilima%2C+B.+et+al.&rft.au=Moyo%2C+K.%3B%2C+Porter%2C+C.%3B+Chilima%2C+B.+et+al.&rft.date=2015&rft.volume=4&rft.issue=1&rft.pages=277&rft_id=info:doi\/10.4102%2Fajlm.v4i1.277&rft_id=info:pmc\/PMC4870597&rft_id=info:pmid\/27213139&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4870597&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PriceLever16-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PriceLever16_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Price, C.P.; John, A.S.; Christenson, R. et al. (2016). \"Leveraging the real value of laboratory medicine with the value proposition\". <i>Clinical Chimica Acta<\/i> <b>462<\/b>: 183\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cca.2016.09.006\" target=\"_blank\">10.1016\/j.cca.2016.09.006<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27649855\" target=\"_blank\">27649855<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Leveraging+the+real+value+of+laboratory+medicine+with+the+value+proposition&rft.jtitle=Clinical+Chimica+Acta&rft.aulast=Price%2C+C.P.%3B+John%2C+A.S.%3B+Christenson%2C+R.+et+al.&rft.au=Price%2C+C.P.%3B+John%2C+A.S.%3B+Christenson%2C+R.+et+al.&rft.date=2016&rft.volume=462&rft.pages=183%E2%80%936&rft_id=info:doi\/10.1016%2Fj.cca.2016.09.006&rft_id=info:pmid\/27649855&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BolognaReduc01-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BolognaReduc01_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bologna, L.; Hardy, G.; Mutter, M. (2001). \"Reducing specimen and medication error with handheld technology\". <i>Proceedings of the 2001 HIMSS Annual Conference<\/i>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reducing+specimen+and+medication+error+with+handheld+technology&rft.jtitle=Proceedings+of+the+2001+HIMSS+Annual+Conference&rft.aulast=Bologna%2C+L.%3B+Hardy%2C+G.%3B+Mutter%2C+M.&rft.au=Bologna%2C+L.%3B+Hardy%2C+G.%3B+Mutter%2C+M.&rft.date=2001&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PlebaniExplor09-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PlebaniExplor09_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Plebani, M. (2009). \"Exploring the iceberg of errors in laboratory medicine\". <i>Clinical Chimica Acta<\/i> <b>404<\/b> (1): 16\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cca.2009.03.022\" target=\"_blank\">10.1016\/j.cca.2009.03.022<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19302995\" target=\"_blank\">19302995<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+the+iceberg+of+errors+in+laboratory+medicine&rft.jtitle=Clinical+Chimica+Acta&rft.aulast=Plebani%2C+M.&rft.au=Plebani%2C+M.&rft.date=2009&rft.volume=404&rft.issue=1&rft.pages=16%E2%80%9323&rft_id=info:doi\/10.1016%2Fj.cca.2009.03.022&rft_id=info:pmid\/19302995&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PlebaniErrors06-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PlebaniErrors06_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Plebani, M. (2006). \"Errors in clinical laboratories or errors in laboratory medicine?\". <i>Clinical Chemistry and Laboratory Medicine<\/i> <b>44<\/b> (6): 750\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2FCCLM.2006.123\" target=\"_blank\">10.1515\/CCLM.2006.123<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16729864\" target=\"_blank\">16729864<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Errors+in+clinical+laboratories+or+errors+in+laboratory+medicine%3F&rft.jtitle=Clinical+Chemistry+and+Laboratory+Medicine&rft.aulast=Plebani%2C+M.&rft.au=Plebani%2C+M.&rft.date=2006&rft.volume=44&rft.issue=6&rft.pages=750%E2%80%939&rft_id=info:doi\/10.1515%2FCCLM.2006.123&rft_id=info:pmid\/16729864&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LewinLab08-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LewinLab08_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-LewinLab08_9-1\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">The Lewin Group (May 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cdc.gov\/labbestpractices\/pdfs\/2007-status-report-laboratory_medicine_-_a_national_status_report_from_the_lewin_group_updated_2008-9.pdf\" target=\"_blank\">\"Laboratory Medicine: A National Status Report\"<\/a> (PDF). Centers for Disease Control and Prevention<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cdc.gov\/labbestpractices\/pdfs\/2007-status-report-laboratory_medicine_-_a_national_status_report_from_the_lewin_group_updated_2008-9.pdf\" target=\"_blank\">https:\/\/www.cdc.gov\/labbestpractices\/pdfs\/2007-status-report-laboratory_medicine_-_a_national_status_report_from_the_lewin_group_updated_2008-9.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Laboratory+Medicine%3A+A+National+Status+Report&rft.atitle=&rft.aulast=The+Lewin+Group&rft.au=The+Lewin+Group&rft.date=May+2008&rft.pub=Centers+for+Disease+Control+and+Prevention&rft_id=https%3A%2F%2Fwww.cdc.gov%2Flabbestpractices%2Fpdfs%2F2007-status-report-laboratory_medicine_-_a_national_status_report_from_the_lewin_group_updated_2008-9.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaronCompu11-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BaronCompu11_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Baron, J.M.; Dighe, A.S. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162747\" target=\"_blank\">\"Computerized provider order entry in the clinical laboratory\"<\/a>. <i>Journal of Pathology Informatics<\/i> <b>2<\/b>: 35. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2F2153-3539.83740\" target=\"_blank\">10.4103\/2153-3539.83740<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3162747\/\" target=\"_blank\">PMC3162747<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21886891\" target=\"_blank\">21886891<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162747\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162747<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Computerized+provider+order+entry+in+the+clinical+laboratory&rft.jtitle=Journal+of+Pathology+Informatics&rft.aulast=Baron%2C+J.M.%3B+Dighe%2C+A.S.&rft.au=Baron%2C+J.M.%3B+Dighe%2C+A.S.&rft.date=2011&rft.volume=2&rft.pages=35&rft_id=info:doi\/10.4103%2F2153-3539.83740&rft_id=info:pmc\/PMC3162747&rft_id=info:pmid\/21886891&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3162747&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PetroseAss16-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PetroseAss16_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-PetroseAss16_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Petrose, L.G.; Fisher, A.M.; Douglas, G.P. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162747\" target=\"_blank\">\"Assessing Perceived Challenges to Laboratory Testing at a Malawian Referral Hospital\"<\/a>. <i>American Journal of Tropical Medicine and Hygiene<\/i> <b>94<\/b> (6): 1426-32. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4269%2Fajtmh.15-0867\" target=\"_blank\">10.4269\/ajtmh.15-0867<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3162747\/\" target=\"_blank\">PMC3162747<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27022150\" target=\"_blank\">27022150<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162747\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162747<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Assessing+Perceived+Challenges+to+Laboratory+Testing+at+a+Malawian+Referral+Hospital&rft.jtitle=American+Journal+of+Tropical+Medicine+and+Hygiene&rft.aulast=Petrose%2C+L.G.%3B+Fisher%2C+A.M.%3B+Douglas%2C+G.P.+et+al.&rft.au=Petrose%2C+L.G.%3B+Fisher%2C+A.M.%3B+Douglas%2C+G.P.+et+al.&rft.date=2016&rft.volume=94&rft.issue=6&rft.pages=1426-32&rft_id=info:doi\/10.4269%2Fajtmh.15-0867&rft_id=info:pmc\/PMC3162747&rft_id=info:pmid\/27022150&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3162747&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VempalaC4G16-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VempalaC4G16_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-VempalaC4G16_12-1\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vempala, S.; Chopra, N.; Rajagopal, A. et al. (2016). \"C4G BLIS: Health Care Delivery via Iterative Collaborative Design in Resource-constrained Settings\". <i>Proceedings of the Eighth International Conference on Information and Communication Technologies and Development<\/i>: 21. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2909609.2909657\" target=\"_blank\">10.1145\/2909609.2909657<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=C4G+BLIS%3A+Health+Care+Delivery+via+Iterative+Collaborative+Design+in+Resource-constrained+Settings&rft.jtitle=Proceedings+of+the+Eighth+International+Conference+on+Information+and+Communication+Technologies+and+Development&rft.aulast=Vempala%2C+S.%3B+Chopra%2C+N.%3B+Rajagopal%2C+A.+et+al.&rft.au=Vempala%2C+S.%3B+Chopra%2C+N.%3B+Rajagopal%2C+A.+et+al.&rft.date=2016&rft.pages=21&rft_id=info:doi\/10.1145%2F2909609.2909657&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DriessenInform15-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DriessenInform15_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-DriessenInform15_13-1\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-DriessenInform15_13-2\">13.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Driessen, J.; Limula, H.; Gadabu, O.J. et al. (2015). \"Informatics solutions for bridging the gap between clinical and laboratory services in a low-resource setting\". <i>African Journal of Laboratory Medicine<\/i> <b>4<\/b> (1): 176. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v4i1.176\" target=\"_blank\">10.4102\/ajlm.v4i1.176<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Informatics+solutions+for+bridging+the+gap+between+clinical+and+laboratory+services+in+a+low-resource+setting&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Driessen%2C+J.%3B+Limula%2C+H.%3B+Gadabu%2C+O.J.+et+al.&rft.au=Driessen%2C+J.%3B+Limula%2C+H.%3B+Gadabu%2C+O.J.+et+al.&rft.date=2015&rft.volume=4&rft.issue=1&rft.pages=176&rft_id=info:doi\/10.4102%2Fajlm.v4i1.176&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CohnUser05-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CohnUser05_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Cohn, M. (2004). <i>User Stories Applied: For Agile Software Development<\/i> (1st ed.). Addison-Wesley Professional. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780321205681.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=User+Stories+Applied%3A+For+Agile+Software+Development&rft.aulast=Cohn%2C+M.&rft.au=Cohn%2C+M.&rft.date=2004&rft.edition=1st&rft.pub=Addison-Wesley+Professional&rft.isbn=9780321205681&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TuthillTheLab14-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TuthillTheLab14_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tuthill, J.M.; Friedman, B.A.; Balis, U.J. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3986538\" target=\"_blank\">\"The laboratory information system functionality assessment tool: Ensuring optimal software support for your laboratory\"<\/a>. <i>Journal of Pathology Informatics<\/i> <b>5<\/b> (1): 7. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2F2153-3539.127819\" target=\"_blank\">10.4103\/2153-3539.127819<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3986538\/\" target=\"_blank\">PMC3986538<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24741466\" target=\"_blank\">24741466<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3986538\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3986538<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+laboratory+information+system+functionality+assessment+tool%3A+Ensuring+optimal+software+support+for+your+laboratory&rft.jtitle=Journal+of+Pathology+Informatics&rft.aulast=Tuthill%2C+J.M.%3B+Friedman%2C+B.A.%3B+Balis%2C+U.J.+et+al.&rft.au=Tuthill%2C+J.M.%3B+Friedman%2C+B.A.%3B+Balis%2C+U.J.+et+al.&rft.date=2014&rft.volume=5&rft.issue=1&rft.pages=7&rft_id=info:doi\/10.4103%2F2153-3539.127819&rft_id=info:pmc\/PMC3986538&rft_id=info:pmid\/24741466&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3986538&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FriedmanEval06-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FriedmanEval06_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Friedman, C.P.; Wyatt, J., ed. (2006). <i>Evaluation Methods in Biomedical Informatics<\/i>. Springer-Verlag. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F0-387-30677-3\" target=\"_blank\">10.1007\/0-387-30677-3<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780387306773.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Evaluation+Methods+in+Biomedical+Informatics&rft.date=2006&rft.pub=Springer-Verlag&rft_id=info:doi\/10.1007%2F0-387-30677-3&rft.isbn=9780387306773&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MonuDesign10-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MonuDesign10_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Monu, R. (27 May 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/hdl.handle.net\/1853\/34792\" target=\"_blank\">\"Design and implementation of a basic laboratory information system for resource-limited settings\"<\/a>. <i>SMARTech: Georgia Tech Theses and Dissertations<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/hdl.handle.net\/1853\/34792\" target=\"_blank\">http:\/\/hdl.handle.net\/1853\/34792<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Design+and+implementation+of+a+basic+laboratory+information+system+for+resource-limited+settings&rft.atitle=SMARTech%3A+Georgia+Tech+Theses+and+Dissertations&rft.aulast=Monu%2C+R.&rft.au=Monu%2C+R.&rft.date=27+May+2010&rft_id=http%3A%2F%2Fhdl.handle.net%2F1853%2F34792&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DouglasUsing10-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DouglasUsing10_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Douglas, G.P.; Gadabu, O.J.; Joukes, S. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2919419\" target=\"_blank\">\"Using touchscreen electronic medical record systems to support and monitor national scale-up of antiretroviral therapy in Malawi\"<\/a>. <i>PLoS Medicine<\/i> <b>7<\/b> (8): e1000319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1000319\" target=\"_blank\">10.1371\/journal.pmed.1000319<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2919419\/\" target=\"_blank\">PMC2919419<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20711476\" target=\"_blank\">20711476<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2919419\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2919419<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+touchscreen+electronic+medical+record+systems+to+support+and+monitor+national+scale-up+of+antiretroviral+therapy+in+Malawi&rft.jtitle=PLoS+Medicine&rft.aulast=Douglas%2C+G.P.%3B+Gadabu%2C+O.J.%3B+Joukes%2C+S.+et+al.&rft.au=Douglas%2C+G.P.%3B+Gadabu%2C+O.J.%3B+Joukes%2C+S.+et+al.&rft.date=2010&rft.volume=7&rft.issue=8&rft.pages=e1000319&rft_id=info:doi\/10.1371%2Fjournal.pmed.1000319&rft_id=info:pmc\/PMC2919419&rft_id=info:pmid\/20711476&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2919419&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MtongaAComp18-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MtongaAComp18_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mtonga, T.; Abaye, M.; Rosko, S.C.; Douglas, G.P. (2018). \"A comparative usability study of two touchscreen clinical workstations for use in low resource settings\". <i>Journal of Health Informatics in Africa<\/i> <b>5<\/b> (2): 78\u201384. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.12856%2FJHIA-2018-v5-i2-209\" target=\"_blank\">10.12856\/JHIA-2018-v5-i2-209<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comparative+usability+study+of+two+touchscreen+clinical+workstations+for+use+in+low+resource+settings&rft.jtitle=Journal+of+Health+Informatics+in+Africa&rft.aulast=Mtonga%2C+T.%3B+Abaye%2C+M.%3B+Rosko%2C+S.C.%3B+Douglas%2C+G.P.&rft.au=Mtonga%2C+T.%3B+Abaye%2C+M.%3B+Rosko%2C+S.C.%3B+Douglas%2C+G.P.&rft.date=2018&rft.volume=5&rft.issue=2&rft.pages=78%E2%80%9384&rft_id=info:doi\/10.12856%2FJHIA-2018-v5-i2-209&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BHBaobabHealth19-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BHBaobabHealth19_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Baobab Health. <a rel=\"nofollow\" class=\"external text\" href=\"#work\">\"Baobab Health Electronic Medical Records Systems Foot Print\"<\/a><span class=\"printonly\">. <a rel=\"nofollow\" class=\"external free\" href=\"#work\">http:\/\/www.baobabhealth.org\/#work<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Baobab+Health+Electronic+Medical+Records+Systems+Foot+Print&rft.atitle=&rft.aulast=Baobab+Health&rft.au=Baobab+Health&rft_id=http%3A%2F%2Fwww.baobabhealth.org%2F%23work&rfr_id=info:sid\/en.wikipedia.org:Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The article mentions \"Online Appendix 1,\" but it could not be found on the journal website. Neither is it clear which GitHub repository applies to their developed software, which the authors claim is on GitHub.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204351\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.525 seconds\nReal time usage: 2.840 seconds\nPreprocessor visited node count: 16664\/1000000\nPreprocessor generated node count: 35097\/1000000\nPost\u2010expand include size: 125899\/2097152 bytes\nTemplate argument size: 41124\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 371.919 1 - -total\n 79.94% 297.302 1 - Template:Reflist\n 67.52% 251.131 20 - Template:Citation\/core\n 55.55% 206.617 15 - Template:Cite_journal\n 14.75% 54.859 1 - Template:Infobox_journal_article\n 14.35% 53.375 1 - Template:Infobox\n 8.83% 32.825 3 - Template:Cite_web\n 8.05% 29.953 32 - Template:Citation\/identifier\n 7.00% 26.018 2 - Template:Cite_book\n 6.87% 25.554 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11807-0!*!0!!en!5!* and timestamp 20200707204349 and revision id 39580\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting\">https:\/\/www.limswiki.org\/index.php\/Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","eb76e424fc635068c27be185a754eac7_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3f\/Tab1_Mtonga_AfricanJofLabMed2019_8-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a5\/Tab2_Mtonga_AfricanJofLabMed2019_8-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/73\/Fig1_Mtonga_AfricanJofLabMed2019_8-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/4c\/Fig2_Mtonga_AfricanJofLabMed2019_8-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/08\/Fig3_Mtonga_AfricanJofLabMed2019_8-1.jpg"],"eb76e424fc635068c27be185a754eac7_timestamp":1594154628,"41e81da03f2f60115a39008f3e196345_type":"article","41e81da03f2f60115a39008f3e196345_title":"From command-line bioinformatics to bioGUI (Joppich and Zimmer 2019)","41e81da03f2f60115a39008f3e196345_url":"https:\/\/www.limswiki.org\/index.php\/Journal:From_command-line_bioinformatics_to_bioGUI","41e81da03f2f60115a39008f3e196345_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:From command-line bioinformatics to bioGUI\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nFrom command-line bioinformatics to bioGUIJournal\n \nPeerJAuthor(s)\n \nJoppich, Markus; Zimmer, RalfAuthor affiliation(s)\n \nLudwig-Maximilians-Universit\u00e4t M\u00fcnchenPrimary contact\n \nEmail: joppich at bio dot ifi dot lmu dot deEditors\n \nGillespie, JosephYear published\n \n2019Volume and issue\n \n7Page(s)\n \ne8111DOI\n \n10.7717\/peerj.8111ISSN\n \n2167-8359Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/peerj.com\/articles\/8111\/Download\n \nhttps:\/\/peerj.com\/articles\/8111.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methods \n\n3.1 Existing workflow systems \n\n3.1.1 Galaxy and Yabi \n3.1.2 Nextflow and DolphinNext \n3.1.3 Common Workflow Language \n3.1.4 SEED 2 and bioinformatics through windows \n3.1.5 RGG and AutoIt \n\n\n3.2 Comparison of existing workflow systems to bioGUI \n3.3 Use case study \n3.4 bioGUI approach \n\n3.4.1 Install modules \n3.4.2 bioGUI templates \n3.4.3 bioGUI integration with CWL and argparse \n\n\n\n\n4 Results \n\n4.1 bioGUI templates \n4.2 Benchmarking bioGUI templates \n\n\n5 Discussion \n\n5.1 Use-case analysis \n5.2 bioGUI repository \n5.3 Availability and extensibility \n5.4 Benchmarking bioGUI templates \n\n\n6 Conclusion \n7 Appendix \n\n7.1 Use cases \n\n7.1.1 Non-computer expert \n7.1.2 Software developers \n\n\n7.2 bioGUI paper mockup \n7.3 Extending templates with script nodes \n7.4 Evaluating a bioGUI template \n7.5 Running programs via bioGUI \n7.6 Hardware specification for benchmarks \n7.7 Template access \n7.8 User survey \n\n\n8 Supplemental information \n9 Acknowledgements \n\n9.1 Authors\u2019 contributions \n9.2 Funding \n9.3 Data availability \n9.4 Competing interests \n\n\n10 References \n11 Notes \n\n\n\nAbstract \nBioinformatics is a highly interdisciplinary field providing informatics applications for scientists from many disciplines. Installing and starting applications on the command line (CL) is inconvenient and inefficient for many scientists. Nonetheless, most methods are implemented with a command-line interface only. Providing a graphical user interface (GUI) for bioinformatics applications is one step toward routinely making CL-only applications more readily available to scientists, yielding a positive step toward more effective interdisciplinary work. With our bioGUI framework, we address two main problems of using CL bioinformatics applications. First, many tools work on UNIX-based systems only, while many scientists use Microsoft Windows. Second, scientists refrain from using CL tools, which, despite their reservations, could well support them in their research. With bioGUI install modules and templates, installing and using CL tools is made possible for most scientists, even on Windows, due to bioGUI\u2019s support for Windows Subsystem for Linux. In addition, bioGUI templates can easily be created, making the bioGUI framework highly rewarding for developers. From the bioGUI repository it is possible to download, install, and use bioinformatics tools with just a few clicks.\n\nIntroduction \nMany advances in bioinformatics rely on sophisticated applications. Examples include Trinity[1] for de novo assembly in conjunction with Trimmomatic[2], or the HISAT2, StringTie, and Ballgown pipeline for transcript-level expression analysis.[3] These tools have in common that when locally installed, only a command-line interface (CLI) is provided, implying a burden for many conducting sequence analysis and alignments who are not computing-adept.[4] Jellyfish[5], Glimmer[6], and HMMER natively run only in UNIX-environments and require a sophisticated setup on Windows. In addition, the installation of command-line (CL) tools is a challenge for non-computer specialists, for example, due to package dependency resolution. This problem has been addressed by the AlgoRun package[7], providing a Docker-based repository of tools. Being a web-based service, it limits use to web-applicable data sizes, or local data must be made available to the Docker container in the cloud. While AlgoRun has the advantage of processing data anywhere, it relies on Docker. Docker may be run either on a local workstation or in the cloud. On a local workstation it can induce incompatibilities with existing software (using Hyper-V on Windows). A cloud-based service may conflict with data privacy guidelines[8], for example, with respect to a possible de-anonymization of patient samples.[9] Using Windows Subsystem for Linux (WSL) is often possible in such a scenario: it is provided as an app from the Microsoft Store.\nA frequent argument for not providing a graphical user interface (GUI) is the overhead for developing it and the effort to make it truly \u201cuser centered.\u201d Often GUIs are simply deemed unnecessary by application developers. However, one can be skeptical whether scientists who are not computing-adept can efficiently use CLIs in their research. In fact, bioinformatician Dr. Istv\u00e1n Albert[10] notes that \u201cBioinformatics, unfortunately, has quite the number of methods that represent the disconnect of the Ivory Tower.\u201d Pavelin et al.[11] note that software is often developed without a focus on usability of interfaces (for end-users). While this does not imply that any GUI is helpful, we argue that without a GUI, the otherwise highly sophisticated CL applications are not very useful for some scientists. Besides, a GUI is often more convenient and helps to avoid using the wrong parameters, especially if an application is not yet routinely used in a lab. University of Western Ontario's David Roy Smith[12] also states that GUI-driven applications make daily work in biology or medical labs easier. Smith remarks that many end-users have a \u201cpenchant for point and click,\u201d not being able to effectively use CL tools. Still, they should have the ability to access and analyse their own data. Many proprietary software solutions address this demand: they allow GUI-based data management, while also being extensible via plug-ins. Smith[13] also points out that one of the biggest advantages of such plugins is to combine the power of peer-reviewed algorithms with a user-friendly GUI. Thus, providing a GUI is an important step toward the applicability of methods by end-users. \nVisne et al.[14] present a universal GUI for R aiming to close the gap between R developers and GUI-dependent users with limited R scripting skills. Additionally, web-based workflow systems, like Galaxy[15] or Yabi[16] provide a means to easily execute bioinformatics applications, but they tend to focus on more complex workflows. However, both Galaxy and Yabi are designed to be run and maintained by bioinformaticians for several users and are not meant to run on a single, individual basis, like in small labs. More recently Morais et al.[4] stated that the accessibility of bioinformatics applications is one of the main challenges of contemporary biology, and that one of the main problems for users is the struggle of using CLIs. While a GUI does not make an application user-friendly per se, it helps to make it more accessible by lowering the burden to use it.[14][4][17][18][19]\nIn recent Microsoft Windows operating systems the WSL feature can be activated. This feature provides a native, non-virtualized Ubuntu environment on Windows, allowing to run most applications that also run on Ubuntu. This solves the problems of running Unix-based tools on Windows. Remaining problems for scientists aiming to run bioinformatics applications thus might be the installation and usage of CL applications.\nHere we present bioGUI, an open-source cross-platform framework for making CL applications more accessible via a GUI. It uses an XML-based domain-specific language (DSL) for template definition, which lowers the initial effort to create a GUI. bioGUI templates for CL applications can easily be scripted. Combined with install modules, the templates provide an efficient and convenient method to deploy bioinformatics applications on Microsoft Windows (via WSL), Mac OS, and Linux. bioGUI also addresses protocol\/parameter management by saving completed templates, enabling easy reproducibility of data analyses (Fig. 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Only little human interaction is needed to run a CL application from a bioGUI template. An (install) template has to be submitted to the bioGUI repository by a developer (blue). The bioGUI application (cyan) allows users (yellow) to download templates or install modules and install and use bioinformatics applications. After the user selected\/set the input for the application using the GUI, the CL arguments to run it are constructed from this input. The application\u2019s output (text or images) can be directly displayed in bioGUI.\n\n\n\nMethods \nThis section first summarizes existing GUI-based systems, then it covers the use-case study we performed and goes into detail of how bioGUI works.\n\nExisting workflow systems \nThere are several workflow systems already available. Most prominent in bioinformatics are the Galaxy server and Yabi. In addition, workflow specification languages such as the Common Workflow Language (CWL) or Nextflow exist. These workflows do not directly compare to bioGUI because they (usually) require a server infrastructure and are not aimed to run on a local computer. However, they have in common that no CLI is needed to run bioinformatics applications.\nWith the R Gui Generator (RGG) a general GUI framework for R already exists. Recently, specialized GUI frameworks like SEED 2[19] or RNA CoMPASS[17] have been presented.\n\nGalaxy and Yabi \nThe Galaxy server[15] is a well known workflow system in bioinformatics. While bioGUI does not aim to be a workflow system like Galaxy\u2014for example, allowing data management\u2014there are similarities. For instance, Galaxy also provides a web-based GUI for its workflows. However, all data to be processed by Galaxy must either be on the server itself or uploaded to a location that is reachable by the server. Galaxy can access cloud storage, but classified data may not be uploaded to such storage, as pointed out in the introduction. Additionally, Galaxy requires Unix knowledge to be installed and does not provide a binary for installation. Galaxy is not cross-platform compatible. (Microsoft Windows is supported through WSL but still requires Unix knowledge.) Galaxy users provide Docker containers for Galaxy, where a local storage can be mounted.\nAnother framework providing similar options is Yabi (Hunter et al., 2012). Yabi is only distributed using a Docker container.\n\nNextflow and DolphinNext \nThe combination of Nextflow and DolphinNext provides similar functionality to Galaxy or Yabi. While Nextflow is a DSL for describing general workflows (lacking a GUI definition), DolphinNext provides the web-based user interface (UI) which enables a convenient usage of Nextflow workflows. Nextflow requires a POSIX system architecture and may or may not run on Microsoft Windows using Cygwin. DolphinNext resembles a lot the Galaxy framework, which can make use of CWL workflows; however, it focuses on a deployment in a cluster environment. It is unknown whether or not both systems work on WSL.\n\nCommon Workflow Language \nThe CWL[20] is a new standard for workflow definition and defines a DSL. In this language, inputs, input-types, and the corresponding parameters are stored. Additionally, inputs can have a help text included.\nUsing the bio.tools ToolDog software[21], CWL workflows can be generated and exported for many bioinformatics applications. An advantage of using bio.tools is the automatic annotation and description of input and outputs. Unfortunately, for many packages no CWL workflows have been deposited.\n\nSEED 2 and bioinformatics through windows \nIn contrast to the previously mentioned tools, SEED 2[19] and Bioinformatics through windows (BTW)[4] do not focus on running complex workflows in a cluster environment. Instead, these focus on specific tasks which can be run on regular laptop computers. SEED 2 focuses on amplicon high-throughput sequencing data analyses. On the other hand, BTW follows the same concept but focuses on the analysis of marker gene data and does not provide a GUI for this task. SEED 2 provides a GUI to perform the relevant analyses fast and conveniently, while BTW focuses on the usability of Unix CL tools on Windows.\n\nRGG and AutoIt \nRGG was developed as a general GUI framework for R applications.[14] It uses XML files to specify the input fields for the graphical representation. When the user has set all options, the GUI is translated into an R script for execution. The execution output can also be retrieved from the RGGRunner application. The RGG software is limited to R scripts, but the authors have expressed their hope that providing GUI for analytical pipelines could \u201chelp to bridge the gap between the R developer community and GUI-dependent users.\u201d[14] \nIn contrast to RGG, AutoIt is a general automation framework which, similar to bioGUI, allows the definition of a GUI as well as a task that is executed according to this input. In contrast to AutoIt, bioGUI is cross-platform compatible, supports WSL, and provides install modules for bioinformatics applications.\n\nComparison of existing workflow systems to bioGUI \nbioGUI is not a classical workflow system like Galaxy, CWL, or DolphinNext, when paired with Nextflow. bioGUI is not meant to run many tasks nor to run in a cluster environment. Moreover, bioGUI does not share the philosophy of having a compute cluster set up to run analyses in a repeated fashion. bioGUI is meant to enable the user to perform bioinformatics analysis at their work place. With bioGUI we aim to provide low-effort usage of bioinformatics applications, without the need to set up a complicated environment. This allows users to easily compare different methods on collected experimental data.\nbioGUI finds its niche as a generalization of the concepts introduced by V\u011btrovsk\u00fd, Baldrian, and Morais[19] and Morais et al.[4] SEED 2 provides a GUI such that a broad public has access to sophisticated and well-known bioinformatics CL applications in the context of amplicon analysis. Similar concepts, yet differently implemented, are provided by RNA CoMPASS[17] for pathogen-host transcriptome analysis or PipeCraft.[18] Here, custom web-based UIs let the user interact with their specialized pipelines. RGG[14] offers a general GUI framework for R applications only. bioGUI offers a similar framework, which is applicable to any Unix application. In both, RGG and bioGUI, users\/developers specify the visual elements in an XML file. This XML file is then interpreted and translated into a GUI within an application (RGGRunner or bioGUI, respectively), which also shows the output of the script.\nThe bioGUI framework extends the concepts presented by RGG and SEED 2, for instance, to general applications, and improves accessibility to these applications by providing install modules.\n\nUse case study \nOne of the main goals we had in mind when developing bioGUI was to create a powerful framework, which is easy-to-use for several types of users and that does not create significant overhead for the application developer. In order to study this, we introduce two classes of possible users. The first class represents a general user of the software who generally prefers a GUI for performing a research task, for example, data analysis after sequencing. The second class describes a software developer releasing an application of a new algorithm to solve the alignment of sequencing reads. This class thus depicts a typical developer.\nFrom these two use cases (see also the Appendix section \u201cUse cases\u201d) we identify the requirements or goals for bioGUI:\n\nInstalling new programs must be simple and should not require system administrators.\nCreating a GUI for a program must not take a lot of time.\nTemplates must bring a basic GUI to run the programs, and output must not be interpreted.\nTemplates must be saveable for later re-use and reference, and also be searchable.\nThe system must be lightweight (runtime overhead, disk-space) to allow for running applications on laptops.\nInstalling a program may require additional (protected) external files.\nFinally, we developed a paper mock-up, with which we went through the anticipated workflow of the user. We identified several input components and features the bioGUI program has to include (Fig. A1, in the Appendix).\n\nbioGUI approach \n\u201cThe accessibility of bioinformatics applications is crucial and a challenge of contemporary biology.\u201d[4] Particularly, the usage of CLIs poses a problem. Since most bioinformatics applications require the execution of commands on the CL for installation (such as for compilation, adding dependencies to the path variable, etc.), we estimate installation also poses a problem.\nDuring the use-case study, and interviews with wet-lab scientists without a computational background (Q. Emslander, 2019, personal communication; L. Jimenez, 2019, personal communication), we found two main problems with bioinformatics applications for scientists, which we want to address with bioGUI: the installation of potentially useful applications and the application's usage. Both problems have in common that they are expected to be performed on the CL. A GUI for achieving the respective tasks in bioinformatics (and beyond) is missing.\nIn particular, the first task of installing bioinformatics applications on a user\u2019s machine poses a few problems. Most bioinformatics applications are written for a Unix operating system, like Linux or Mac OS, while in general Microsoft Windows is the dominant operating system. In order to overcome this problem, bioGUI makes use of WSL on Windows. Even if the user\u2019s OS is already Unix-like, using the CL to install software might prove cumbersome. Thus, in order to support all users, bioGUI uses a cross-platform approach. bioGUI is developed in C++ using the Qt framework.\nThe general workflow for any program using bioGUI is shown in Fig. 1 (above). Given a CL application, the software developer (blue) writes the specific template in an XML-based DSL and can then make this template available, for example, in the bioGUI repository (cyan). Such templates can be automatically retrieved by bioGUI. Upon selection of a template by the user, bioGUI displays the input mask as defined in the template. When the user (yellow) has filled in all parameters, the parameters are collected by bioGUI and assembled into CL arguments, which are used to execute the original CL-only application. Upon completion, simple results (like text-output or images) can be shown in bioGUI directly or within an external application that is opened.\n\nInstall modules \nInstall modules are designed to install applications such that bioGUI can access them. Essentially, install modules are bash scripts which allow an automatic installation of applications into a predefined location. For this purpose, install modules receive several arguments from bioGUI when launched, for example, where to install the application to, the sudo password to fetch packages via a system\u2019s package manager (e.g., aptitude, conda), whether the application should be made available to the user via the system\u2019s PATH variable, etc. Install modules download and install applications and make them available to the user and bioGUI. However, some applications cannot be simply downloaded, but are distributed by installers. For this purpose, the install module template can be extended by further input fields. These must be specified by bioGUI elements, and their values are added to the end of the CL arguments of the install module. An install module can then execute the referenced installer.\nFinally, an install module should contain the specification of its bioGUI template and could hardcode the path to the installed application. Other constant values, which can already be derived during the installation (e.g., absolute paths to dependencies), could also be defined in the template during this stage.\n\nbioGUI templates \nbioGUI templates are the actual end-user-interface to programs. A bioGUI template defines the look and functions of the UI. Thus it can define how the CL-application is called (with corresponding parameters).\nEach bioGUI template consists of two parts (Fig. A2). The first part (the <window> model) defines the visual appearance of the GUI. The second part (the <execution> model) defines the processing logic of the template. Input values from the GUI components are collected and assembled (e.g., pre-\/post-processing steps) to call CL applications. As part of this assembly, input values from the GUI may be transformed using multiple predefined nodes. Concatenations are possible using the <add> node, and constant values can be inserted using the <const> node. System environment properties, such as the operating system, the computer\u2019s IP address, or specific directories can be collected using the <env> node. If the regular nodes are not sufficient, for example, because more complex string manipulations should be made (see use-case study), script nodes may also accept functions written in LUA or JavaScript.\nIn general, the execution part infers a network with inputs (e.g., GUI elements, other nodes within the execution part) and actions (if, add). For example, the execution network for an application with many sub-commands is exemplarily shown in Fig. A3 (see Appendix).\nThe time to template varies with the application as well as the number of options to be included. A simple template, like the one for MS-EmpiRe[22], can be created within 10 minutes. More comprehensive templates, like the one for HISAT2, usually take about 30 minutes. Time can be saved if only the most important command line options are shown in the GUI. This can be achieved by adding an \u201coptional parameters\u201d input field, where users can insert CL arguments themselves. This is, for instance, shown in the wtdbg2[23] and spades[24] templates. Adding the install part to a template usually can be done within 15 to 30 minutes, depending on how detailed the build process is documented. The creation of an install module thus takes approximately one hour.\n\nbioGUI integration with CWL and argparse \nThe CWL[20] only describes the CL workflow and neither provides a GUI nor a means to install the desired tool. Due to this more general specification, CWL fits most problems, but specific annotations of inputs, explanations, or the embedding of images is not supported in CWL.\nWhile developers can always create templates manually, bioGUI supports developers by offering a template generator from CWL templates or python3 argparse CL parsers. Since there are already many CWL templates available for bioinformatics CL applications, CWL files can be used as a base to automatically generate bioGUI templates from. Using the bioGUI template generator for argparse, it is also possible to automatically generate templates from CWL files (making use of the cwl2argparse program provided by CWL). Our generator takes as input the argparse parser or CWL file and creates input elements for all elements. In case the type of an input is unclear or not supported, the generator falls back to a regular text-input element.\n\nResults \nbioGUI templates \nCurrently more than 25 install modules exist for bioGUI. These represent basically three groups of bioinformatics tasks: next-generation sequencing data analysis and transcriptomics, long read sequencing analysis and assembly, and a more general sequence analysis. In general, these install modules will install the respective application on the local machine. The Circlator[25] template allows to pull and use the corresponding Docker image. The available tools, as well as their respective categorization, are listed in Table 1.\n\n\n\n\n\n\n\nTable 1. List of available templates and install modules (starting with Install) for bioGUI; tools marked with \u2713 provide an install module for the operating system of the respective column.\n\n\n\nModule name\n\nTask\n\nInstall module\n\n\nWSL and Ubuntu\n\nMac OS\n\n\nFirst Time Mac OS Setup\n\nInitialization\n\n\u2013\n\n\u2713\n\n\nFirst Time Ubuntu\/WSL\/apt-get Setup\n\nInitialization\n\n\u2713\n\n\u2013\n\n\nInstall Ballgown v1.0.1[3]\n\nNGS transcriptomics\n\n\u2713\n\n\n\n\nInstall Bowtie1[26]\n\nNGS\n\n\u2713\n\n\n\n\nInstall Bowtie2 v2.2.9[27]\n\nNGS\n\n\u2713\n\n\u2713\n\n\nInstall bwa v0.7.17[28]\n\nNGS\n\n\u2713\n\n\u2713\n\n\nInstall canu[29]\n\nAssembly\n\n\u2713\n\n\n\n\nInstall featureCounts[30]\n\nNGS transcriptomics\n\n\u2713\n\n\u2713\n\n\nInstall glimmer302b[6]\n\nGenome annotation\n\n\u2713\n\n\n\n\nInstall graphmap[31]\n\nLong read sequencing\n\n\u2713\n\n\u2713\n\n\nInstall albacore (pip wheel, ONT)\n\nLong read sequencing\n\n\u2713\n\n\n\n\nInstall guppy (linux tar.gz, ONT)\n\nLong read sequencing\n\n\u2713\n\n\n\n\nInstall hisat2[32]\n\nNGS transcriptomics\n\n\u2713\n\n\u2713\n\n\nInstall hmmer-3.1b2[33]\n\nSequence analysis\n\n\u2713\n\n\n\n\nInstall jellyfish-2.2.6[5]\n\nNGS\n\n\u2713\n\n\n\n\nInstall minimap2\/miniasm\/racon (gitHub)\n\nAssembly (long-read)\n\n\u2713\n\n\u2713\n\n\nInstall MS-EmpiRe[22]\n\nNGS transcriptomics\n\n\u2713\n\n\u2713\n\n\nInstall PureSeqTM[34]\n\nSequence analysis\n\n\u2713\n\n\n\n\nInstall rMATS-3.2.5[35]\n\nNGS transcriptomics\n\n\u2713\n\n\n\n\nInstall rnahybrid[36]\n\nSequence analysis\n\n\u2713\n\n\u2713\n\n\nInstall RSEM v1.3.0[37]\n\nNGS transcriptomics\n\n\u2713\n\n\n\n\nInstall samtools-1.3.1[38]\n\nNGS\n\n\u2713\n\n\u2713\n\n\nInstall SPAdes v3.13.0[24]\n\nAssembly (hybrid)\n\n\u2713\n\n\u2713\n\n\nInstall StringTie v1.3.0[3]\n\nNGS transcriptomics\n\n\u2713\n\n\n\n\nInstall Top Monitor (ssh example)\n\nTechnical demo\n\n\u2713\n\n\u2713\n\n\nInstall Trimmomatic v0.36[2]\n\nNGS\n\n\u2713\n\n\n\n\nInstall wtdbg2[23]\n\nAssembly (long-read)\n\n\u2713\n\n\n\n\nInstall Circulator[25]\n\nAssembly\n\n\u2713\n\n\u2713\n\n\n\nBenchmarking bioGUI templates \nOur benchmark comprises of four tasks. The first task is to assemble a bacterial genome from Oxford Nanopore long reads, for which the minimap2[39]\/miniasm[40]\/racon[41] pipeline (available as install module from bioGUI) is used. The second task is the quantification of reads from a yeast mRNA sequencing project using Oxford Nanpore Reads and Illumina Reads (EMBL ENA studies PRJNA398797 [MinION] and SAMN00849440 [Illumina]). The quantification is performed using featureCounts from the subread package.[30] The third task uses these results to compute differential gene expression. Differential gene expression analysis is performed using MS-EmpiRe in R (install module available[22]). Finally the fourth task uses RNAhybrid[36] to predict miRNA binding sites (1,978 murine miRNAs) in 170 sequences of each 200 nt.\nThe results are shown in Table 2. The given run times are wall clock times. The peak RAM consumption has been sampled from the process viewer on the given operating systems (Task Manager on Windows, top on Linux and Mac OS).\n\n\n\n\n\n\n\nTable 2. Benchmarking results for the four selected tasks (see Benchmarking bioGUI templates within the Results section) on the described hardware (see Table A1 in Appendix section). All runs are started via bioGUI.\n\n\n\nTask\n\nLinux server\n\nLenovo server\n\nSurface Book\n\nMacBook Air\n\n\nTime\n\nPeak RAM\n\nTime\n\nPeak RAM\n\nTime\n\nPeak RAM\n\nTime\n\nPeak RAM\n\n\nAssembly\n\n10:12 min\n\n6.8 GB\n\n23:00 min\n\n6.5 GB\n\n30:00 min\n\n6.5 GB\n\n44:30 min\n\n6.5 GB\n\n\nfeatureCounts (MinION)\n\n00:38 min\n\n20 MB\n\n00:54 min\n\n18 MB\n\n01:12 min\n\n18 MB\n\n01:30 min\n\n18 MB\n\n\nfeatureCounts (Illumina)\n\n01:13 min\n\n28 MB\n\n01:41 min\n\n25 MB\n\n02:22 min\n\n25 MB\n\n02:30 min\n\n25 MB\n\n\nDE quantification (MinION)\n\n00:19 min\n\n0.7 GB\n\n00:25 min\n\n0.6 GB\n\n00:28 min\n\n0.6 GB\n\n00:42 min\n\n0.6 GB\n\n\nDE quantification (Illumina)\n\n00:14 min\n\n0.5 GB\n\n00:19 min\n\n0.4 GB\n\n00:20 min\n\n0.4 GB\n\n00:31 min\n\n0.4 GB\n\n\nRNAhybrid\n\n07:35 min\n\n19 MB\n\n23:00 min\n\n18 MB\n\n13:00 min\n\n18 MB\n\n16:55 min\n\n18 MB\n\n\n\nDiscussion \nbioGUI is a framework for easy GUI-based usage of CL applications in the life sciences. Using bioGUI, high-quality CL applications can be made accessible to as many researchers as possible. This is achieved by lowering the hurdles to overcome for using bioinformatics applications, particularly on Windows.\n\nUse-case analysis \nOur use-case analysis (Appendix section \u201cUse cases\u201d) has revealed several requirements for bioGUI (see the section \u201cMethods\u201d) to enable the user to perform the sequencing analysis and to allow the developer to rapidly create a template (Fig. 2).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. bioGUI use-case study, from a developer\u2019s and user\u2019s perspective, performed on an exemplary RNAseq analysis workflow. The dark-gray underlayed tasks represent the developer\u2019s tasks, and the bright-yellow part represents the analysis pipeline the user wants to execute. Tasks requiring user-action are shown as rectangles and intermediate results are shown in ellipses. Cyan ellipses denote solutions\/results (e.g., template repository) offered by bioGUI. bioGUI starts sub-processes for each task, such that the overhead for any started processes is as small as possible. Upon finishing a task or pipeline, bioGUI can display a notification and open generated output.\n\n\n\nAn easy installation (goal 1) is given through the availability of install modules, which can be downloaded from the bioGUI repository and started via the GUI. These also allow additional inputs (e.g., Python wheels for albacore, goal 6).\nThe install modules combine the installation of an application and the creation of the actual GUI template. If the developers employ automatic testing of their software (e.g., build checks with Travis), the install part resembles a Travis container setup (goal 2): dependencies and the application itself are installed into an operating system. Even if not, most bioinformaticians extensively use Ubuntu and\/or bash scripts. Thus writing a script to install dependencies is not a significantly hard workload. We have reached a seamless and time-efficient creation of templates using an XML-based DSL. XML is particularly helpful as it allows to specify hierarchies and attributes to objects. Using our template generator for CWL and python3-argparse, bioGUI templates can be created even faster (goal 2). The templates are highly flexible in the creation of CL parameters, also due to providing script nodes. By providing install modules and templates, high-quality open-source bioinformatics applications become more accessible to the community.\nThe bioGUI application is cross-platform compatible and only requires a few megabytes of disk space (goal 5). bioGUI implements several possibilities to execute applications (see Fig. A4, in the Appendix). In general, the only runtime overhead involved is the creation of a bash process, which starts the actual program with the assembled CL arguments (goal 5). bioGUI, being a local stand-alone application, has the possibility to target both locally installed and web-based applications, reachable within a controlled environment and with large data. In addition, bioGUI also supports the use of Docker containers, for cases where all other options fail.\nThe UI can be made easily understandable (goal 3). Using text labels, the user can get help on inputs (if specified by the developer), links can be used to provide further information, and, most importantly, tooltips could also hint the user to which information is needed at a certain step.\nFinally, completed templates can be saved via the \"Save Template\" button in bioGUI, and all available templates can be filtered (goal 4). This enables a user to keep track of performed analyses, and it makes results more reproducible because parameters are saved. Having the possibility to save templates also allows users to easily repeat an analysis with the same parameters. Additionally, using the bioGUI repository, templates can easily be shared among users, making it easier to standardize runs among different users or even institutes.\nAn anonymous survey (with 10 participants) about common problems in using bioinformatics software was also conducted among colleagues (n = 4) and undergraduate students or collaborators from the life sciences (n = 6, short: collaborators). The results are available in the Appendix section.\nWe asked \u201cWhat were the most cumbersome tasks in accessing and using the software?\u201d, referring to recently used bioinformatics software by the participants. Eight of the 10 participants answered that finding parameters or using the software was cumbersome. This shows that the selected goals for bioGUI address actual problems faced by both experts and regular users. We further asked the participants to install and use graphmap[31], which was selected because it is reasonably easy to install and use. First the participants were asked to install the tool using the CLI as well as via bioGUI. For this task, all instructions have been provided. The question \u201cHas the installation process been easy?\u201d (0 = No, 5 = Yes) has been answered with an average score of 4.4 for the CLI and 4.8 for bioGUI. Then the participants were asked to align the given reads against a given reference genome, without giving the instructions. Again we asked \u201cHas it been easy to align the reads?\u201d (0 = No, 5 = Yes). Here the CLI scored a 3.5 on average and bioGUI a 4.9. (Fig. A6). This coincidences with the answer to our question \u201cOverall: Which interface was easier to use in your opinion?\u201d (0 = CLI, 5 = GUI). Here the average score was 4. Bioinformaticians and collaborators answered differently: the average bioinformatician was undecided on which interface was easier to use (average 3), but non-bioinformaticians preferred the GUI over the CLI (average 4.5).\nThe survey indicates that there are problems with bioinformatics software regarding installation and usage of CLI tools. These problems can be reduced by providing a GUI for these programs. The more experienced a user is on the CL, the less impact a GUI has. But particularly for non-experts on the CL, a GUI makes it easier to use a program.\n\nbioGUI repository \nWe provide a repository of preconfigured templates on our website (see Fig. A5, in the Appendix), where authors and users can search for and browse existing templates or submit new ones. bioGUI can access uploaded templates and save them directly for use. Install modules are provided specifically for WSL and Ubuntu users, which manages dependency resolution and installs applications (locally) onto the user\u2019s device. This currently works in any environment using the aptitude (apt) package manager, but users can submit templates which also support other environments, since install modules are versatile bash scripts. On Mac OS, some install modules support Homebrew for template installation. Install modules download and may severely alter a system (especially if the sudo password is supplied). Thus, submitted install modules are manually curated and are only accessible when no security threat has been identified.\nThe major goal of bioGUI is to enable any scientist to use bioinformatics applications. While we extend the repository on a regular basis depending on our own use, users can also request new templates for applications relevant to them.\n\nAvailability and extensibility \nbioGUI is open-source software, and users or institutions can either use our global bioGUI repository or deploy a custom repository, for example, one which is only reachable within an institution.\nbioGUI is available on GitHub. Both source code as well as pre-built binary distributions (for Microsoft Windows, Linux, and Mac OS) are available. While bioGUI will run on any Linux distribution, install modules currently use mainly aptitude as a package manager (e.g., Ubuntu, debian-based distributions). If used on Windows, the same applies for the used WSL-application (Ubuntu 18.04 is recommended). bioGUI has been tested on Microsoft Windows 10, build 17763. On Mac OS, bioGUI uses Homebrew to install dependencies. Homebrew does not support a silent, non-interactive installation: the user has to install Homebrew before running the \"First time setup for Mac OS\" install module, which will then install the most common dependencies.\nWhile a number of use cases and corresponding components are already included in bioGUI, we encourage users to contribute on GitHub by either pushing their own extensions or opening feature requests. Further documentation (installation and setup guide, how to write templates) is also available via ReadTheDocs.\n\nBenchmarking bioGUI templates \nbioGUI starts a subprocess for each executed program. Thus, the only overhead created by bioGUI itself is the one for running the GUI, which creates less than 1% CPU usage, allocates less than 50 MB, and only performs IO operations when loading a new template (assessed via Sysinternals Process Explorer [Microsoft Sysinternals, 2019]).\nNonetheless, we have been interested in demonstrating that many bioinformatics tasks do not require a dedicated server setup but can be performed on regular laptop computers. We thus benchmarked four typical tasks performed using bioGUI.\nThe selected tasks allow a good overview of different demands: Tasks 1 (assembly) and 3 (differential expression analysis) are CPU-bound tasks, while tasks 2 (feature counting) and 4 (miRNA target prediction) are IO-bound. Particularly, Task 2 has a high load of read operations, and Task 4 has a high demand of write operations. We compare these tasks on a dedicated Linux server, one rather powerful Lenovo T470p laptop computer, one Surface Book laptop computer (resource-wise a typical laboratory laptop), and one MacBook Air. The computer specifications are listed in the Table A1 (see Appendix) and results are shown in Table 2 (above). Even though we have not included the alignment of the Illumina yeast reads in this benchmark, it should be noted that this task also runs well on laptop computers. On the Lenovo laptop, the alignment of the SRR453566 sample, consisting of 5,725,730 paired reads, has a peak RAM consumption of 34 MB and took 13:50 minutes, while the Surface Book is even faster at less than eight minutes. This presumably can be explained due to different SSD speeds.\nThe results in Table 2 (above) show that even computationally low-end computers can run bioinformatics tools. More interestingly, these results show that the WSL allows the execution of interesting bioinformatic tools. It can be seen that WSL is slow for IO operations, but it has a comparable speed for in-memory operations. Particularly, tools requiring a lot of IO are considerably faster on the Linux Server (Assembly, RNAhybrid), while the computationally expensive tools like MS-EmpiRe[22] and featureCounts[30] run within similar times.\n\nConclusion \nThe bioGUI framework makes it easy to develop, provide, and use GUIs for CL applications. Particularly for non-computer experts, using CLIs is cumbersome. Providing a GUI and\/or install modules increases accessibility to high-quality bioinformatics applications for these users. bioGUI creates a cross-platform GUI experience for many open-source bioinformatics applications. In particular, bioGUI enables the deployment of academic bioinformatics applications to Microsoft Windows workstations and laptops, as well as Linux or Mac OS.\nThe separation of the GUI components and the program logic allows for the creation of templates in two steps. First, the template developer adds input elements to the window and, second, assembles these inputs according to the needs of the application back into CL arguments. This way almost any CL application can be used with a GUI, enabling many more researchers to use open-source tools. Providing install modules to make Unix applications available to Microsoft Windows users (via WSL) supports this goal.\nbioGUI can not always replace dedicated GUIs. A tailored UI will still be more usable and user-friendly than any generic solution can be. We experienced this in our use case: certain tasks (e.g., selecting options) require special solutions, let alone from displaying or interpreting the results. However, especially with the install module concept, we aim to provide a seamless installation and create the possibility to run CL applications by all scientists. Using the bioGUI framework, simple GUIs can be constructed. But these simple GUIs already help to make bioinformatic tools more accessible by making execution and usage of these tools more comfortable.\nUsing bioGUI, it becomes a simple exercise to use supported CL applications from a GUI. Currently, there are already installed modules and templates for more than 25 applications in our bioGUI repository. bioGUI lowers the burden to use excellent applications, allowing more scientists a better analysis of their data. With bioGUI, it is not necessary to understand how to use and navigate on the CL; instead, the focus is set on the applications, its method and parameters, and finally the data.\n\nAppendix \nUse cases \nNon-computer expert \nMany researchers work in small labs without any significant IT support. The computers in their labs mostly run Microsoft Windows, and PhD students often have to bring their own devices (because the institute does not provide such working devices). Particularly in the life sciences, users can profit greatly from existing open-source software. However, installing major bioinformatics applications on such lab computers often poses a problem: administrators (if existent) have little time to deploy new applications, or there is no support in installing new applications at all. If the users are not computer experts, installing and using command line (CL) tools may be cumbersome for them (see the user surveys in this appendix). While there are users that can use the CL efficiently, the cited literature and our personal experience shows that there are many users who do not feel comfortable on the CL. This does not mean that they don\u2019t want to learn it or are incapable of learning it, but their focus simply does not lie in learning to use the CL. Instead, they want to get the results for their data fast, reliably, and without issue. One of these users is Luisa.\nOxford Nanopore Sequencing is becoming more and more popular, and even the sequencing hardware can be found in more and more biological laboratories, like in Luisa\u2019s. Particularly important for MinION sequencing is the post-processing of the actual raw read data. While in previous versions base-calling was directly performed in the cloud by Oxford Nanopore, this has now been pushed back to the client side. Thus, despite having the sequencing data on her laptop, Luisa must still retrieve the sequences herself, using, for instance, the Albacore basecaller (if they don\u2019t want to rely on LiveBasecalling). Unfortunately, like many bioinformatics packages, the basecaller only comes as a python CL program. Additionally, the download is only available as a Python wheel, which means there is no UI-based setup available. Luisa thus needs assistance for the installation of the Python wheel, as well as starting the basecalling process. After the reads are basecalled, reads need to be aligned to a reference genome. While reference genomes in a correct format exist on her lab computer (or can easily be downloaded from the web), the CL program to map the reads is available only from GitHub to be installed from source. Luisa has trouble using the CL to clone the repository, compile, and use the CL application.\nLuisa does not require a custom analysis of her data but rather wants to initially screen her data in a simple, basic, and robust analysis. She is mostly busy in her lab, hence an analysis has to be prepared fast, and parameters should be stored for later reference. For this, a local searchable database of saved templates is needed.\n\nSoftware developers \nA developer finished his sequence alignment program. The project is already published on GitHub and in a journal, but only a few people start using it. From the issues and feature requests on GitHub it can be seen that mainly other bioinformaticians use the program. Thus, the developer decides that the program should be accessible to more researchers and looks for ways to make the program usable by everyone. Since the program is written in Python, it is cross-platform compatible. However, it is noticed that domain experts do not install and use the program. Thus the developer must look for an easy way to distribute the application and make it accessible to more researchers. The developer\u2019s time is limited, having other projects waiting. There is also little support for developing a GUI from colleagues, as they have different views on the extent of autonomy a wet lab researcher should have regarding sequencing analysis.\n\nbioGUI paper mockup \n\n\n\n\n\n\n\n\n\n Figure A1. bioGUI mockup showing the elements a template could be made of. The GUI has a searchable list of installed templates as well as a link to our repository of templates. The right side is reserved to the currently displayed GUI template. Here a structured view of the available parameters, as well as hints for filling these, is shown to the user. Finally, the user has the possibility to run the program by clicking a button and to see the program\u2019s output.\n\n\n\nExtending templates with script nodes \nOften it is required to perform string manipulations (e.g., remove file extensions) for CL arguments. For instance, the example below takes as input a HISAT2 index file and removes the file extension, such that the index will be accepted by HISAT2. For evaluation of this node, the evaluate function is called with the argv references as input parameters. The last return value of the script\u2019s call stack is taken as the output value of the script -node.\nListing 1. bioGUI script node with LUA function example. Upon evaluating this node, the evaluate function will be called with the arguments listed in the argv attribute of the script node.\n<script id=\u201chisat_index_rel\u201d argv=\u201c${hisat_index_rel_raw}\u201d>\n<\/p><p><![CDATA[\n<\/p><p>function evaluate(arg1)\n<\/p><p>\u2003\u2003\u2003\u2003if (string.match(arg1, \u201c.%d.ht2$\u201d)) then\n<\/p><p>\u2003\u2003\u2003\u2003\u2003\u2003return(string.sub(arg1, 0, arg1:find(\u201c.%d.ht2$\u201d)-1))\n<\/p><p>\u2003\u2003\u2003\u2003end\n<\/p><p>\u2003\u2003\u2003return(arg1)\n<\/p><p>\u2003\u2003end\n<\/p><p>]]>\n<\/p><p><\/script>\n\nEvaluating a bioGUI template \nIn Fig. A2, the process of assembling a CL call from the shown bioGUI template is explained. First, the creation of the <window> model (dark gray) will be explained, followed by the creation of the CL arguments using the <execution> model (shaded).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure A2. Template construction and evaluation in bioGUI. First, the dark gray window part is evaluated to create the GUI. Once the user clicks the run button, the execution part of the template (shaded) is executed by constructing and starting the assembled system call. This system call is constructed in three steps by replacing variables with evaluated terms from the user\u2019s input. Blue lines indicate the visual element a returned value (cyan lines) is taken from. Helper\/intermediate nodes to be evaluated are shown in light yellow.\n\n\n\nThe window component consists of four different components, which are grouped in a vertical layout (default for window component). A label describing the input file dialog is placed on the main window, followed by the actual file dialog with ID input. Then a group box with title and a checkable status is created, which contains an output file dialog. Finally, the action button, which starts the CL call assembly and the subprocess of this program, and the text output elements are created.\nWhen the user has entered all desired data and clicks the action button, the execution phase defined by the execution model will be launched. Therefore the program defined in the execute element is started. For this, the parameters (param) must be assembled. Any text within ${var} is interpreted as a reference to a variable \"var\" or the value of a GUI element with id \"var.\" Thus, the CL is successively assembled. At first the ${input} element is interpreted and retrieves the value from the input file dialog as this element matches the id. Next the ${output} is interpreted. The ${output} refers to an \"if\" construct in the execution part, which compares the value of the element with id \"os\" to the string \"TRUE\" (which indicated whether the groupbox is checked). If this value is true, this node evaluates to netcat 192.168.1.100 55025, otherwise to tee -a {output file path}. Finally, the program \"sh\" is executed with the created CL arguments. For instance, if the group box is checked, the sh -c \u201ccat inputFile | netcat 192.168.1.100 55025\u201d will be executed. A full reference of all input types as well as all execution nodes is available online.\nIn fact, the evaluation of the execution network resembles the simulation of a petri net (Fig. A3). Each node in the execution network is a place, and its modification\/function is the transition, which requires values for all its input places, to generate the output token.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure A3. (A) An automatically generated bioGUI template from the poreSTAT (Internal tool for minION sequencing analysis) python argument parser. (B) The resulting execution network for the bioGUI template shown in (A). The central node represents the fully assembled CL argument (yellow).\n\n\n\nRunning programs via bioGUI \nProgram execution via bioGUI can be accomplished via different paths, which are shown in Fig. A4. The easiest way is to execute a native program (one that runs natively on the operating system, e.g., Docker). Then all output can be piped to bioGUI to display this to the user. If the host is a Microsoft Windows 10 OS, bioGUI can also run Unix programs via WSL. Then the Unix program runs natively in a WSL bash. The resulting output can be transferred to bioGUI via pipes. Of course, for both native and WSL processes, the output can also be transferred via netcat to bioGUI. The transfer of the GUI template within install modules is an example. If a process runs on a remote computer, the output can be transferred to bioGUI also via network, for example, netcat. Such a process can, for instance, be started by calling ssh from bioGUI with appropriate parameters. Finally bioGUI can also send HTTP POST requests to web services and accepts an HTTP response as answer. This output can also be displayed by bioGUI.\nSince the Docker engine is a local, native process, bioGUI also supports the use of Docker containers. The Circlator template is an example of how this can be implemented.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure A4. Possibilities for running bioGUI: locally via processes, on a network via ssh or on the web via HTTP request\/response. Straight arrow (purple): HTTP execution mode; Dotted arrow (green): Docker execution; Dotdashed arrow(orange): bash\/WSL execution; Dashed arrow(cyan): remote\/ssh execution.\n\n\n\nHardware specification for benchmarks \nThe relevant hardware for benchmarking bioGUI is summarized in Table A1.\n\n\n\n\n\n\n\nTable A1.Hardware used to benchmark bioGUI\n\n\n\nComputer name\n\nCPU\n\nRAM\n\nStorage\n\n\nLinux server\n\nIntel Xeon W-2145 CPU @ 3.70 GHz\r\n8 cores (+8 HT cores)\n\n128 GB\n\nSamsung SSD 860, 1 TB SSD\n\n\nLenovo laptop (T470p)\n\nIntel Core i7-7820HQ @ 2.9 GHz\r\n4 cores (+4 HT cores)\n\n32 GB\n\nSamsung MZVLB1T0HALR, 1 TB SSD\n\n\nMicrosoft surface book\n\nIntel Core i5-6300U @ 2.4 GHz\r\n2 cores (+2 HT cores)\n\n8 GB\n\nSamsung MZFLV128HCGR, 128 GB SSD\n\n\nApple MacBook Air (mid 2012)\n\nIntel Core i5 @ 1.7 GHz\n\n8 GB\n\n128 GB SSD\n\n\n\nTemplate access \n\n\n\n\n\n\n\n\n\n Figure A5. (A) On our website a list of already existing templates can be browsed. Besides the description and author, also the type (install module or template) is shown. (B) All uploaded templates can be downloaded directly from within bioGUI. bioGUI allows to search in\/filter all available install modules and templates.\n\n\n\nUser survey \nA user survey with 10 participants (four bioinformaticians, and six collaborators consisting of two undergraduate bioinformatics students and four external collaborators) was performed. The derived results are shown in Table A2, and the raw data are shown in Table A3.\n\n\n\n\n\n\n\nTable A2. Derived user survey results from the given answers\n\n\n\n\n\nn\n\nMedian\n\nMean\n\np-value\n\nVariance\n\n\nBetter interface bio\n\n3\n\n3\n\n3\n\n\n\n4\n\n\nBetter interface collab\n\n6\n\n4.5\n\n4.5\n\n\n\n0.3\n\n\nBetter interface all\n\n9\n\n4\n\n4.00\n\n\n\n1.75\n\n\nEasy to align CLI\n\n10\n\n3.5\n\n3.50\n\n\n\n1.833\n\n\nEasy to align bioGUI\n\n10\n\n5\n\n4.90\n\n0.0098\n\n0.1\n\n\nEasy to install CLI\n\n10\n\n5\n\n4.40\n\n\n\n0.711\n\n\nEasy to install bioGUI\n\n10\n\n5\n\n4.80\n\n0.2023\n\n0.178\n\n\n\n\n\n\n\n\n\nTable A3. Relevant participant answers for the performed user survey and the results in Table A2\n\n\n\n\n\nParticipant 1\n\nParticipant 2\n\nParticipant 3\n\nParticipant 4\n\nParticipant 5\n\nParticipant 6\n\nParticipant 7\n\nParticipant 8\n\nParticipant 9\n\nParticipant 10\n\n\nUsertype (0 = bioinformatician, 1 = student, 2 = collaborator)\n\n0\n\n0\n\n0\n\n0\n\n1\n\n2\n\n2\n\n1\n\n2\n\n2\n\n\nWhich kind of user-interface does the tool have?\n\nCLI\n\nCLI\n\nCLI\n\nCLI\n\nCLI\n\nCLI\n\nGUI\n\nCLI\n\nCLI\n\nGUI\n\n\nWhat were the most cumbersome tasks in accessing and using the software?\n\nDependencies\n\nUsing software, finding settings\n\nFinding settings, options needed\n\nInstalling software\n\nFinding settings, options needed\n\nDependencies, starting the software, using the software\n\nFinding settings, options needed\n\nFinding settings, options needed\n\nUsing the software\n\nFinding settings, options needed\n\n\nCLI: Has the installation process been easy? (0 = NO, 5 = YES)\n\n4\n\n3\n\n5\n\n5\n\n5\n\n5\n\n4\n\n5\n\n5\n\n3\n\n\nCLI: Has it been easy to align the reads? (0 = NO, 5 = YES)\n\n4\n\n2\n\n5\n\n5\n\n4\n\n3\n\n3\n\n5\n\n3\n\n1\n\n\nbioGUI: Has the installation process been easy? (0 = NO, 5 = YES)\n\n5\n\n5\n\n5\n\n4\n\n5\n\n5\n\n5\n\n5\n\n5\n\n4\n\n\nbioGUI: Has it been easy to align the reads? (0 = NO, 5 = YES)\n\n5\n\n5\n\n5\n\n5\n\n5\n\n5\n\n5\n\n5\n\n5\n\n4\n\n\nOverall: Which interface was easier to use in your opinion? (0 = CLI, 5 = GUI)\n\n5\n\n1\n\n3\n\n4\n\n5\n\n5\n\n4\n\n4\n\n5\n\n\n\n\n\n\r\n\n\n\n\n\n\n\n\n\n\n Figure A6. Scores given by the 10 participants on the question \u201cHas it been easy to align the reads?\u201d after performing the task using the CLI and bioGUI. These results show that most participants found the task easier using bioGUI, but for no-one it was harder to use bioGUI.\n\n\n\nSupplemental information \n DOI 10.7717\/peerj.8111\/supp-1 - Survey questions on command-line tools and bioGUI: This is the original survey used to assess problems with current bioinformatics applications. (PDF)\n DOI 10.7717\/peerj.8111\/supp-2 - Answers on the survey on command-line tools and bioGUI: Each column represents a single participant. Questions are in rows. (XLXS)\nAcknowledgements \nWe thank Luisa F. Jimenez-Soto and Gergely Csaba for their valuable input as well as for reviewing the manuscript. We thank the participants in our survey for their time. We thank the reviewers for their constructive feedback.\n\nAuthors\u2019 contributions \nMarkus Joppich conceived and designed the experiments, performed the experiments, analyzed the data, prepared figures and\/or tables, authored or reviewed drafts of the paper, and approved the final draft. Ralf Zimmer conceived and designed the experiments, analyzed the data, prepared figures and\/or tables, authored or reviewed drafts of the paper, and approved the final draft.\n\nFunding \nThis work was supported by the Deutsche Forschungsgemeinschaft (Collaborative Research Centre SFB 1123-2\/Z2). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.\n\nData availability \nThe bioGUI documentation is available here. In order to set up Windows Subsystem for Linux (required for using bioGUI on Windows), follow the steps documented here. bioGUI is open-source software. Releases and code are available on the GitHub project page. Additional software (cwl2biogui) is available here.\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nReferences \n\n\n\u2191 Grabherr, M.G.; Haas, B.J.; Yassour, M. et al. (2011). \"Full-length transcriptome assembly from RNA-Seq data without a reference genome\". Nature Biotechnology 29 (7): 644\u201352. doi:10.1038\/nbt.1883. PMC PMC3571712. PMID 21572440. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3571712 .   \n\n\u2191 2.0 2.1 Bolger, A.M.; Lohse, M.; Usadel, B. (2014). \"Trimmomatic: A flexible trimmer for Illumina sequence data\". Bioinformatics 30 (15): 2114-20. doi:10.1093\/bioinformatics\/btu170. PMC PMC4103590. PMID 24695404. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4103590 .   \n\n\u2191 3.0 3.1 3.2 Pertea, M.; Kim, D.; Pertea, G.M. et al. (2016). Transcript-level expression analysis of RNA-seq experiments with HISAT, StringTie and Ballgown. 11. pp. 1650\u201367. doi:10.1038\/nprot.2016.095. PMC PMC5032908. PMID 27560171. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5032908 .   \n\n\u2191 4.0 4.1 4.2 4.3 4.4 4.5 Morais, D.; Roesch, L.F.W.; Redmile-Gordon, M. et al. (2018). BTW-Bioinformatics Through Windows: An easy-to-install package to analyze marker gene data. 6. pp. e5299. doi:10.7717\/peerj.5299. PMC PMC6074753. PMID 30083449. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6074753 .   \n\n\u2191 5.0 5.1 Mar\u00e7ais, G.; Kingsford, C. (2011). \"A fast, lock-free approach for efficient parallel counting of occurrences of k-mers\". Bioinformatics 27 (6): 764\u201370. doi:10.1093\/bioinformatics\/btr011. PMC PMC3051319. PMID 21217122. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3051319 .   \n\n\u2191 6.0 6.1 Delcher, A.L.; Bratke, K.A.; Powers, E.C. et al. (2007). \"Identifying bacterial genes and endosymbiont DNA with Glimmer\". Bioinformatics 23 (6): 673\u20139. doi:10.1093\/bioinformatics\/btm009. PMC PMC2387122. PMID 17237039. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2387122 .   \n\n\u2191 Hosny, A.; Vera-Licona, P.; Laubenbacher, R. et al. (2016). \"AlgoRun: A Docker-based packaging system for platform-agnostic implemented algorithms\". Bioinformatics 32 (15): 2396\u20138. doi:10.1093\/bioinformatics\/btw120. PMC PMC6280798. PMID 27153722. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6280798 .   \n\n\u2191 Schadt, E.E. (2012). \"The changing privacy landscape in the era of big data\". Molecular Systems Biology 8: 612. doi:10.1038\/msb.2012.47. PMC PMC3472686. PMID 22968446. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3472686 .   \n\n\u2191 Gymrek, M.; McGuire, A.L.; Golan, D. et al. (2013). \"Identifying personal genomes by surname inference\". Science 339 (6117): 321\u20134. doi:10.1126\/science.1229566. PMID 23329047.   \n\n\u2191 Albert, I. (2016). \"The Biostar Handbook\". https:\/\/biostar.myshopify.com\/ .   \n\n\u2191 Pavelin, K.; Cham, J.A.; de Matos, P. et al. (2012). \"Bioinformatics meets user-centred design: a perspective\". PLoS Computational Biology 8 (7): e1002554. doi:10.1371\/journal.pcbi.1002554. PMC PMC3395592. PMID 22807660. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3395592 .   \n\n\u2191 Smith, D.R. (2013). \"The battle for user-friendly bioinformatics\". Frontiers in Genetics 4: 187. doi:10.3389\/fgene.2013.00187. PMC PMC3778374. PMID 24065986. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3778374 .   \n\n\u2191 Smith, D.R. (2015). Buying in to bioinformatics: An introduction to commercial sequence analysis software. 16. p. 700-9. doi:10.1093\/bib\/bbu030. PMC PMC4501248. PMID 25183247. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4501248 .   \n\n\u2191 14.0 14.1 14.2 14.3 14.4 Visne, I.; Dilaveroglu, E.; Vierlinger, K. et al. (2009). RGG: A general GUI Framework for R scripts. 10. p. 74. doi:10.1186\/1471-2105-10-74. PMC PMC2653488. PMID 19254356. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2653488 .   \n\n\u2191 15.0 15.1 Afgan, E.; Baker, D.; van den Beek, M. et al. (2016). \"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update\". Nucleic Acids Research 44 (W1): W3\u2013W10. doi:10.1093\/nar\/gkw343. PMC PMC4987906. PMID 27137889. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987906 .   \n\n\u2191 Hunter, A.A.; Macgregor, A.B.; Szabo, T.O. et al. (2012). \"Yabi: An online research environment for grid, high performance and cloud computing\". Source Code for Biology and Medicine 7 (1): 1. doi:10.1186\/1751-0473-7-1. PMC PMC3298538. PMID 22333270. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3298538 .   \n\n\u2191 17.0 17.1 17.2 Xu, G.; Strong, M.J.; Lacey, M.R. et al. (2014). \"RNA CoMPASS: A dual approach for pathogen and host transcriptome analysis of RNA-seq datasets\". PLoS One 9 (2): e89445. doi:10.1371\/journal.pone.0089445. PMC PMC3934900. PMID 24586784. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3934900 .   \n\n\u2191 18.0 18.1 Anslan, S.; Bahram, M.; Hiirsalu, I. et al. (2017). \"PipeCraft: Flexible open-source toolkit for bioinformatics analysis of custom high-throughput amplicon sequencing data\". Molecular Ecology Resources 17 (6): e234-e240. doi:10.1111\/1755-0998.12692. PMID 28544559.   \n\n\u2191 19.0 19.1 19.2 19.3 Vetrovsk\u00fd, T.; Baldrian, P.; Morais, D. (2018). \"SEED 2: A user-friendly platform for amplicon high-throughput sequencing data analyses\". Bioinformatics 34 (13): 2292-2294. doi:10.1093\/bioinformatics\/bty071. PMC PMC6022770. PMID 29452334. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6022770 .   \n\n\u2191 20.0 20.1 Amstutz, P.; Andeer, R.; Chapman, B. et al. (15 March 2016). \"Common Workflow Language, Draft 3\". FigShare. https:\/\/figshare.com\/articles\/Common_Workflow_Language_draft_3\/3115156\/1 .   \n\n\u2191 Hillion KH1, Kuzmin I2, Khodak A. et al. (2017). \"Using bio.tools to generate and annotate workbench tool descriptions\". F1000Research 6: ELIXIR-2074. doi:10.12688\/f1000research.12974.1. PMC PMC5747335. PMID 29333231. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5747335 .   \n\n\u2191 22.0 22.1 22.2 22.3 Ammar, C.; Gruber, M.; Csaba, G. et al. (2019). \"MS-EmpiRe Utilizes Peptide-level Noise Distributions for Ultra-sensitive Detection of Differentially Expressed Proteins\". Molecular and Cellular Proteomics 18 (9): 1880\u201392. doi:10.1074\/mcp.RA119.001509. PMC PMC6731086. PMID 31235637. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6731086 .   \n\n\u2191 23.0 23.1 Ruan, J.; Li, H. (2019). \"Fast and accurate long-read assembly with wtdbg2\". Nature Methods. doi:10.1038\/s41592-019-0669-3. PMID 31819265.   \n\n\u2191 24.0 24.1 Bankevich, A.; Nurk, S.; Antipov, D. et al. (2012). \"SPAdes: A new genome assembly algorithm and its applications to single-cell sequencing\". Journal of Computational Biology. doi:10.1089\/cmb.2012.0021. PMC PMC3342519. PMID 22506599. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3342519 .   \n\n\u2191 25.0 25.1 Hunt, M.; Silva, N.D.; Otto, T.D. et al. (2015). \"Circlator: Automated circularization of genome assemblies using long sequencing reads\". Genome Biology 16: 294. doi:10.1186\/s13059-015-0849-0. PMC PMC4699355. PMID 26714481. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4699355 .   \n\n\u2191 Langmead, B.; Trapnell, C.; Pop, M. et al. (2009). \"Ultrafast and memory-efficient alignment of short DNA sequences to the human genome\". Genome Biology 10 (3): R25. doi:10.1186\/gb-2009-10-3-r25. PMC PMC2690996. PMID 19261174. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2690996 .   \n\n\u2191 Langmead, B.; Salzberg, S.L. (2012). \"Fast gapped-read alignment with Bowtie 2\". Mature Methods 9 (4): 357\u20139. doi:10.1038\/nmeth.1923. PMC PMC3322381. PMID 22388286. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3322381 .   \n\n\u2191 Li, H.; Durbin, R. (2009). \"Fast and accurate short read alignment with Burrows-Wheeler transform\". Bioinformatics 25 (14): 1754\u201360. doi:10.1093\/bioinformatics\/btp324. PMC PMC2705234. PMID 19451168. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234 .   \n\n\u2191 Koren, S.; Walenz, B.P.; Berlin, K. et al. (2017). \"Canu: Scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation\". Genome Research 27 (5): 722\u201336. doi:10.1101\/gr.215087.116. PMC PMC5411767. PMID 28298431. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5411767 .   \n\n\u2191 30.0 30.1 30.2 Liao, Y.; Smyth, G.K.; Shi, W. (2014). \"featureCounts: an efficient general purpose program for assigning sequence reads to genomic features\". Bioinformatics 30 (7): 923\u201330. doi:10.1093\/bioinformatics\/btt656. PMID 24227677.   \n\n\u2191 31.0 31.1 Sovi\u0107, I.; \u0160iki\u0107, M.; Wilm, A. et al. (2016). \"Fast and sensitive mapping of nanopore sequencing reads with GraphMap\". Nature Communications 7: 11307. doi:10.1038\/ncomms11307. PMC PMC4835549. PMID 27079541. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4835549 .   \n\n\u2191 Kim, D.; Paggi, J.M.; Park, C. et al. (2019). \"Graph-based genome alignment and genotyping with HISAT2 and HISAT-genotype\". Nature Biotechnology 37 (8): 907\u201315. doi:10.1038\/s41587-019-0201-4. PMID 31375807.   \n\n\u2191 Wheeler, T.J.; Eddy, S.R. (2013). \"nhmmer: DNA homology search with profile HMMs\". Bioinformatics 29 (19): 2487-9. doi:10.1093\/bioinformatics\/btt403. PMC PMC3777106. PMID 23842809. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3777106 .   \n\n\u2191 Wang, Q.; Ni, C.-m., Li, Z. et al. (2019). \"PureseqTM: efficient and accurate prediction of transmembrane topology from amino acid sequence only\". bioRxiv. doi:10.1101\/627307.   \n\n\u2191 Shen, S.; Park, J.W.; Lu, Z.X. et al. (2014). \"rMATS: Robust and flexible detection of differential alternative splicing from replicate RNA-Seq data\". Proceedings of the National Academy of Sciences of the United States of America 111 (51). doi:10.1073\/pnas.1419161111. PMC PMC4280593. PMID 25480548. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4280593 .   \n\n\u2191 36.0 36.1 Rehmsmeier, M.; Steffen, P.; Hochsmann, M. et al. (2004). \"Fast and effective prediction of microRNA\/target duplexes\". RNA 10 (10). doi:10.1261\/rna.5248604. PMC PMC4280593. PMID 15383676. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4280593 .   \n\n\u2191 Li, B.; Dewey, C.N. (323). \"RSEM: Accurate transcript quantification from RNA-Seq data with or without a reference genome\". BMC Bioinformatics 12. doi:10.1186\/1471-2105-12-323. PMC PMC3163565. PMID 21816040. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163565 .   \n\n\u2191 Li, H.; Handsaker, B.; Wysoker, A. et al. (2009). \"The Sequence Alignment\/Map format and SAMtools\". Bioinformatics 25 (16): 2078-9. doi:10.1093\/bioinformatics\/btp352. PMC PMC2723002. PMID 19505943. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002 .   \n\n\u2191 Li, H. (2018). \"Minimap2: pairwise alignment for nucleotide sequences\". Bioinformatics 34 (18): 3094\u2013100. doi:10.1093\/bioinformatics\/bty191. PMC PMC6137996. PMID 29750242. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6137996 .   \n\n\u2191 Li, H. (2016). \"Minimap and miniasm: Fast mapping and de novo assembly for noisy long sequences\". Bioinformatics 32 (14): 2103\u201310. doi:10.1093\/bioinformatics\/btw152. PMC PMC4937194. PMID 27153593. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4937194 .   \n\n\u2191 Vaser, R.; Sovi\u0107, I.; Nagarajan, N. et al. (2017). \"Fast and accurate de novo genome assembly from long uncorrected reads\". Genome Research 27 (5): 737-46. doi:10.1101\/gr.214270.116. PMC PMC5411768. PMID 28100585. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5411768 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:From_command-line_bioinformatics_to_bioGUI\">https:\/\/www.limswiki.org\/index.php\/Journal:From_command-line_bioinformatics_to_bioGUI<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 7 January 2020, at 00:53.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 636 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","41e81da03f2f60115a39008f3e196345_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_From_command-line_bioinformatics_to_bioGUI skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:From command-line bioinformatics to bioGUI<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">Bioinformatics<\/a> is a highly interdisciplinary field providing informatics applications for scientists from many disciplines. Installing and starting applications on the command line (CL) is inconvenient and inefficient for many scientists. Nonetheless, most methods are implemented with a command-line interface only. Providing a graphical user interface (GUI) for bioinformatics applications is one step toward routinely making CL-only applications more readily available to scientists, yielding a positive step toward more effective interdisciplinary work. With our bioGUI framework, we address two main problems of using CL bioinformatics applications. First, many tools work on UNIX-based systems only, while many scientists use Microsoft Windows. Second, scientists refrain from using CL tools, which, despite their reservations, could well support them in their research. With bioGUI install modules and templates, installing and using CL tools is made possible for most scientists, even on Windows, due to bioGUI\u2019s support for Windows Subsystem for Linux. In addition, bioGUI templates can easily be created, making the bioGUI framework highly rewarding for developers. From the bioGUI repository it is possible to download, install, and use bioinformatics tools with just a few clicks.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Many advances in <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> rely on sophisticated applications. Examples include Trinity<sup id=\"rdp-ebb-cite_ref-GrabherrFull11_1-0\" class=\"reference\"><a href=\"#cite_note-GrabherrFull11-1\">[1]<\/a><\/sup> for <i>de novo<\/i> assembly in conjunction with Trimmomatic<sup id=\"rdp-ebb-cite_ref-BolgerTrimmo14_2-0\" class=\"reference\"><a href=\"#cite_note-BolgerTrimmo14-2\">[2]<\/a><\/sup>, or the HISAT2, StringTie, and Ballgown pipeline for transcript-level expression analysis.<sup id=\"rdp-ebb-cite_ref-PerteaTranscript16_3-0\" class=\"reference\"><a href=\"#cite_note-PerteaTranscript16-3\">[3]<\/a><\/sup> These tools have in common that when locally installed, only a command-line interface (CLI) is provided, implying a burden for many conducting sequence analysis and alignments who are not computing-adept.<sup id=\"rdp-ebb-cite_ref-MoraisBTW18_4-0\" class=\"reference\"><a href=\"#cite_note-MoraisBTW18-4\">[4]<\/a><\/sup> Jellyfish<sup id=\"rdp-ebb-cite_ref-Mar.C3.A7aisAFast11_5-0\" class=\"reference\"><a href=\"#cite_note-Mar.C3.A7aisAFast11-5\">[5]<\/a><\/sup>, Glimmer<sup id=\"rdp-ebb-cite_ref-DelcherIdent07_6-0\" class=\"reference\"><a href=\"#cite_note-DelcherIdent07-6\">[6]<\/a><\/sup>, and <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/hmmer.org\/\" target=\"_blank\">HMMER<\/a> natively run only in UNIX-environments and require a sophisticated setup on Windows. In addition, the installation of command-line (CL) tools is a challenge for non-computer specialists, for example, due to package dependency resolution. This problem has been addressed by the AlgoRun package<sup id=\"rdp-ebb-cite_ref-HosnyAlgoRun16_7-0\" class=\"reference\"><a href=\"#cite_note-HosnyAlgoRun16-7\">[7]<\/a><\/sup>, providing a Docker-based repository of tools. Being a web-based service, it limits use to web-applicable data sizes, or local data must be made available to the Docker container in the cloud. While AlgoRun has the advantage of processing data anywhere, it relies on Docker. Docker may be run either on a local workstation or in the cloud. On a local workstation it can induce incompatibilities with existing software (using Hyper-V on Windows). A <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud-based<\/a> service may conflict with <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_privacy\" title=\"Information privacy\" class=\"wiki-link\" data-key=\"185f6d9f874e48914b5789317408f782\">data privacy<\/a> guidelines<sup id=\"rdp-ebb-cite_ref-SchadtTheChang12_8-0\" class=\"reference\"><a href=\"#cite_note-SchadtTheChang12-8\">[8]<\/a><\/sup>, for example, with respect to a possible de-anonymization of patient samples.<sup id=\"rdp-ebb-cite_ref-GymrekIdent13_9-0\" class=\"reference\"><a href=\"#cite_note-GymrekIdent13-9\">[9]<\/a><\/sup> Using Windows Subsystem for Linux (WSL) is often possible in such a scenario: it is provided as an app from the Microsoft Store.\n<\/p><p>A frequent argument for not providing a graphical user interface (GUI) is the overhead for developing it and the effort to make it truly \u201cuser centered.\u201d Often GUIs are simply deemed unnecessary by application developers. However, one can be skeptical whether scientists who are not computing-adept can efficiently use CLIs in their research. In fact, bioinformatician Dr. Istv\u00e1n Albert<sup id=\"rdp-ebb-cite_ref-AlberBiostar16_10-0\" class=\"reference\"><a href=\"#cite_note-AlberBiostar16-10\">[10]<\/a><\/sup> notes that \u201cBioinformatics, unfortunately, has quite the number of methods that represent the disconnect of the Ivory Tower.\u201d Pavelin <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-PavelinBioinfo12_11-0\" class=\"reference\"><a href=\"#cite_note-PavelinBioinfo12-11\">[11]<\/a><\/sup> note that software is often developed without a focus on usability of interfaces (for end-users). While this does not imply that any GUI is helpful, we argue that without a GUI, the otherwise highly sophisticated CL applications are not very useful for some scientists. Besides, a GUI is often more convenient and helps to avoid using the wrong parameters, especially if an application is not yet routinely used in a lab. University of Western Ontario's David Roy Smith<sup id=\"rdp-ebb-cite_ref-SmithTheBattle13_12-0\" class=\"reference\"><a href=\"#cite_note-SmithTheBattle13-12\">[12]<\/a><\/sup> also states that GUI-driven applications make daily work in biology or medical labs easier. Smith remarks that many end-users have a \u201cpenchant for point and click,\u201d not being able to effectively use CL tools. Still, they should have the ability to access and analyse their own data. Many proprietary software solutions address this demand: they allow GUI-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a>, while also being extensible via plug-ins. Smith<sup id=\"rdp-ebb-cite_ref-SmithBuying15_13-0\" class=\"reference\"><a href=\"#cite_note-SmithBuying15-13\">[13]<\/a><\/sup> also points out that one of the biggest advantages of such plugins is to combine the power of peer-reviewed algorithms with a user-friendly GUI. Thus, providing a GUI is an important step toward the applicability of methods by end-users. \n<\/p><p>Visne et al.<sup id=\"rdp-ebb-cite_ref-VisneRGG09_14-0\" class=\"reference\"><a href=\"#cite_note-VisneRGG09-14\">[14]<\/a><\/sup> present a universal GUI for <a href=\"https:\/\/www.limswiki.org\/index.php\/R_(programming_language)\" title=\"R (programming language)\" class=\"wiki-link\" data-key=\"1b0aa598f071aca4c5b4ee08d8bb2bde\">R<\/a> aiming to close the gap between R developers and GUI-dependent users with limited R scripting skills. Additionally, web-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a> systems, like <a href=\"https:\/\/www.limswiki.org\/index.php\/Galaxy_(biomedical_software)\" title=\"Galaxy (biomedical software)\" class=\"wiki-link\" data-key=\"ead5d6ebaa8d67744d2f68d454d89ce6\">Galaxy<\/a><sup id=\"rdp-ebb-cite_ref-AfganTheGal16_15-0\" class=\"reference\"><a href=\"#cite_note-AfganTheGal16-15\">[15]<\/a><\/sup> or Yabi<sup id=\"rdp-ebb-cite_ref-HunterYabi12_16-0\" class=\"reference\"><a href=\"#cite_note-HunterYabi12-16\">[16]<\/a><\/sup> provide a means to easily execute bioinformatics applications, but they tend to focus on more complex workflows. However, both Galaxy and Yabi are designed to be run and maintained by bioinformaticians for several users and are not meant to run on a single, individual basis, like in small labs. More recently Morais <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MoraisBTW18_4-1\" class=\"reference\"><a href=\"#cite_note-MoraisBTW18-4\">[4]<\/a><\/sup> stated that the accessibility of bioinformatics applications is one of the main challenges of contemporary biology, and that one of the main problems for users is the struggle of using CLIs. While a GUI does not make an application user-friendly per se, it helps to make it more accessible by lowering the burden to use it.<sup id=\"rdp-ebb-cite_ref-VisneRGG09_14-1\" class=\"reference\"><a href=\"#cite_note-VisneRGG09-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MoraisBTW18_4-2\" class=\"reference\"><a href=\"#cite_note-MoraisBTW18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-XuRNA14_17-0\" class=\"reference\"><a href=\"#cite_note-XuRNA14-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AnslanPipeCraft17_18-0\" class=\"reference\"><a href=\"#cite_note-AnslanPipeCraft17-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Vetrovsk.C3.BDSeed18_19-0\" class=\"reference\"><a href=\"#cite_note-Vetrovsk.C3.BDSeed18-19\">[19]<\/a><\/sup>\n<\/p><p>In recent Microsoft Windows operating systems the WSL feature can be activated. This feature provides a native, non-virtualized Ubuntu environment on Windows, allowing to run most applications that also run on Ubuntu. This solves the problems of running Unix-based tools on Windows. Remaining problems for scientists aiming to run bioinformatics applications thus might be the installation and usage of CL applications.\n<\/p><p>Here we present bioGUI, an open-source cross-platform framework for making CL applications more accessible via a GUI. It uses an XML-based domain-specific language (DSL) for template definition, which lowers the initial effort to create a GUI. bioGUI templates for CL applications can easily be scripted. Combined with install modules, the templates provide an efficient and convenient method to deploy bioinformatics applications on Microsoft Windows (via WSL), Mac OS, and Linux. bioGUI also addresses protocol\/parameter management by saving completed templates, enabling easy reproducibility of data analyses (Fig. 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Joppich_PeerJ2019_7.jpg\" class=\"image wiki-link\" data-key=\"2d7d34bfe7dfc294159bd95b9f80b915\"><img alt=\"Fig1 Joppich PeerJ2019 7.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/1f\/Fig1_Joppich_PeerJ2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Only little human interaction is needed to run a CL application from a bioGUI template. An (install) template has to be submitted to the bioGUI repository by a developer (blue). The bioGUI application (cyan) allows users (yellow) to download templates or install modules and install and use bioinformatics applications. After the user selected\/set the input for the application using the GUI, the CL arguments to run it are constructed from this input. The application\u2019s output (text or images) can be directly displayed in bioGUI.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>This section first summarizes existing GUI-based systems, then it covers the use-case study we performed and goes into detail of how bioGUI works.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Existing_workflow_systems\">Existing workflow systems<\/span><\/h3>\n<p>There are several workflow systems already available. Most prominent in bioinformatics are the Galaxy server and Yabi. In addition, workflow specification languages such as the Common Workflow Language (CWL) or Nextflow exist. These workflows do not directly compare to bioGUI because they (usually) require a server infrastructure and are not aimed to run on a local computer. However, they have in common that no CLI is needed to run bioinformatics applications.\n<\/p><p>With the R Gui Generator (RGG) a general GUI framework for R already exists. Recently, specialized GUI frameworks like SEED 2<sup id=\"rdp-ebb-cite_ref-Vetrovsk.C3.BDSeed18_19-1\" class=\"reference\"><a href=\"#cite_note-Vetrovsk.C3.BDSeed18-19\">[19]<\/a><\/sup> or RNA CoMPASS<sup id=\"rdp-ebb-cite_ref-XuRNA14_17-1\" class=\"reference\"><a href=\"#cite_note-XuRNA14-17\">[17]<\/a><\/sup> have been presented.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Galaxy_and_Yabi\">Galaxy and Yabi<\/span><\/h4>\n<p>The Galaxy server<sup id=\"rdp-ebb-cite_ref-AfganTheGal16_15-1\" class=\"reference\"><a href=\"#cite_note-AfganTheGal16-15\">[15]<\/a><\/sup> is a well known workflow system in bioinformatics. While bioGUI does not aim to be a workflow system like Galaxy\u2014for example, allowing data management\u2014there are similarities. For instance, Galaxy also provides a web-based GUI for its workflows. However, all data to be processed by Galaxy must either be on the server itself or uploaded to a location that is reachable by the server. Galaxy can access cloud storage, but classified data may not be uploaded to such storage, as pointed out in the introduction. Additionally, Galaxy requires Unix knowledge to be installed and does not provide a binary for installation. Galaxy is not cross-platform compatible. (Microsoft Windows is supported through WSL but still requires Unix knowledge.) Galaxy users provide Docker containers for Galaxy, where a local storage can be mounted.\n<\/p><p>Another framework providing similar options is Yabi (Hunter et al., 2012). Yabi is only distributed using a Docker container.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Nextflow_and_DolphinNext\">Nextflow and DolphinNext<\/span><\/h4>\n<p>The combination of <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nextflow.io\/\" target=\"_blank\">Nextflow<\/a> and <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/UMMS-Biocore\/dolphinnext\" target=\"_blank\">DolphinNext<\/a> provides similar functionality to Galaxy or Yabi. While Nextflow is a DSL for describing general workflows (lacking a GUI definition), DolphinNext provides the web-based user interface (UI) which enables a convenient usage of Nextflow workflows. Nextflow requires a POSIX system architecture and may or may not run on Microsoft Windows using <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/cygwin.com\/index.html\" target=\"_blank\">Cygwin<\/a>. DolphinNext resembles a lot the Galaxy framework, which can make use of CWL workflows; however, it focuses on a deployment in a cluster environment. It is unknown whether or not both systems work on WSL.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Common_Workflow_Language\">Common Workflow Language<\/span><\/h4>\n<p>The CWL<sup id=\"rdp-ebb-cite_ref-AmstutzCommon16_20-0\" class=\"reference\"><a href=\"#cite_note-AmstutzCommon16-20\">[20]<\/a><\/sup> is a new standard for workflow definition and defines a DSL. In this language, inputs, input-types, and the corresponding parameters are stored. Additionally, inputs can have a help text included.\n<\/p><p>Using the bio.tools ToolDog software<sup id=\"rdp-ebb-cite_ref-HillionUsing17_21-0\" class=\"reference\"><a href=\"#cite_note-HillionUsing17-21\">[21]<\/a><\/sup>, CWL workflows can be generated and exported for many bioinformatics applications. An advantage of using bio.tools is the automatic annotation and description of input and outputs. Unfortunately, for many packages no CWL workflows have been deposited.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"SEED_2_and_bioinformatics_through_windows\">SEED 2 and bioinformatics through windows<\/span><\/h4>\n<p>In contrast to the previously mentioned tools, SEED 2<sup id=\"rdp-ebb-cite_ref-Vetrovsk.C3.BDSeed18_19-2\" class=\"reference\"><a href=\"#cite_note-Vetrovsk.C3.BDSeed18-19\">[19]<\/a><\/sup> and Bioinformatics through windows (BTW)<sup id=\"rdp-ebb-cite_ref-MoraisBTW18_4-3\" class=\"reference\"><a href=\"#cite_note-MoraisBTW18-4\">[4]<\/a><\/sup> do not focus on running complex workflows in a cluster environment. Instead, these focus on specific tasks which can be run on regular laptop computers. SEED 2 focuses on amplicon high-throughput sequencing data analyses. On the other hand, BTW follows the same concept but focuses on the analysis of marker gene data and does not provide a GUI for this task. SEED 2 provides a GUI to perform the relevant analyses fast and conveniently, while BTW focuses on the usability of Unix CL tools on Windows.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"RGG_and_AutoIt\">RGG and AutoIt<\/span><\/h4>\n<p>RGG was developed as a general GUI framework for R applications.<sup id=\"rdp-ebb-cite_ref-VisneRGG09_14-2\" class=\"reference\"><a href=\"#cite_note-VisneRGG09-14\">[14]<\/a><\/sup> It uses XML files to specify the input fields for the graphical representation. When the user has set all options, the GUI is translated into an R script for execution. The execution output can also be retrieved from the RGGRunner application. The RGG software is limited to R scripts, but the authors have expressed their hope that providing GUI for analytical pipelines could \u201chelp to bridge the gap between the R developer community and GUI-dependent users.\u201d<sup id=\"rdp-ebb-cite_ref-VisneRGG09_14-3\" class=\"reference\"><a href=\"#cite_note-VisneRGG09-14\">[14]<\/a><\/sup> \n<\/p><p>In contrast to RGG, <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.autoitscript.com\/site\/autoit\/\" target=\"_blank\">AutoIt<\/a> is a general automation framework which, similar to bioGUI, allows the definition of a GUI as well as a task that is executed according to this input. In contrast to AutoIt, bioGUI is cross-platform compatible, supports WSL, and provides install modules for bioinformatics applications.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Comparison_of_existing_workflow_systems_to_bioGUI\">Comparison of existing workflow systems to bioGUI<\/span><\/h3>\n<p>bioGUI is not a classical workflow system like Galaxy, CWL, or DolphinNext, when paired with Nextflow. bioGUI is not meant to run many tasks nor to run in a cluster environment. Moreover, bioGUI does not share the philosophy of having a compute cluster set up to run analyses in a repeated fashion. bioGUI is meant to enable the user to perform bioinformatics analysis at their work place. With bioGUI we aim to provide low-effort usage of bioinformatics applications, without the need to set up a complicated environment. This allows users to easily compare different methods on collected experimental data.\n<\/p><p>bioGUI finds its niche as a generalization of the concepts introduced by V\u011btrovsk\u00fd, Baldrian, and Morais<sup id=\"rdp-ebb-cite_ref-Vetrovsk.C3.BDSeed18_19-3\" class=\"reference\"><a href=\"#cite_note-Vetrovsk.C3.BDSeed18-19\">[19]<\/a><\/sup> and Morais <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MoraisBTW18_4-4\" class=\"reference\"><a href=\"#cite_note-MoraisBTW18-4\">[4]<\/a><\/sup> SEED 2 provides a GUI such that a broad public has access to sophisticated and well-known bioinformatics CL applications in the context of amplicon analysis. Similar concepts, yet differently implemented, are provided by RNA CoMPASS<sup id=\"rdp-ebb-cite_ref-XuRNA14_17-2\" class=\"reference\"><a href=\"#cite_note-XuRNA14-17\">[17]<\/a><\/sup> for pathogen-host transcriptome analysis or PipeCraft.<sup id=\"rdp-ebb-cite_ref-AnslanPipeCraft17_18-1\" class=\"reference\"><a href=\"#cite_note-AnslanPipeCraft17-18\">[18]<\/a><\/sup> Here, custom web-based UIs let the user interact with their specialized pipelines. RGG<sup id=\"rdp-ebb-cite_ref-VisneRGG09_14-4\" class=\"reference\"><a href=\"#cite_note-VisneRGG09-14\">[14]<\/a><\/sup> offers a general GUI framework for R applications only. bioGUI offers a similar framework, which is applicable to any Unix application. In both, RGG and bioGUI, users\/developers specify the visual elements in an XML file. This XML file is then interpreted and translated into a GUI within an application (RGGRunner or bioGUI, respectively), which also shows the output of the script.\n<\/p><p>The bioGUI framework extends the concepts presented by RGG and SEED 2, for instance, to general applications, and improves accessibility to these applications by providing install modules.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Use_case_study\">Use case study<\/span><\/h3>\n<p>One of the main goals we had in mind when developing bioGUI was to create a powerful framework, which is easy-to-use for several types of users and that does not create significant overhead for the application developer. In order to study this, we introduce two classes of possible users. The first class represents a general user of the software who generally prefers a GUI for performing a research task, for example, data analysis after sequencing. The second class describes a software developer releasing an application of a new algorithm to solve the alignment of sequencing reads. This class thus depicts a typical developer.\n<\/p><p>From these two use cases (see also the Appendix section \u201cUse cases\u201d) we identify the requirements or goals for bioGUI:\n<\/p>\n<ol><li>Installing new programs must be simple and should not require system administrators.<\/li>\n<li>Creating a GUI for a program must not take a lot of time.<\/li>\n<li>Templates must bring a basic GUI to run the programs, and output must not be interpreted.<\/li>\n<li>Templates must be saveable for later re-use and reference, and also be searchable.<\/li>\n<li>The system must be lightweight (runtime overhead, disk-space) to allow for running applications on laptops.<\/li>\n<li>Installing a program may require additional (protected) external files.<\/li><\/ol>\n<p>Finally, we developed a paper mock-up, with which we went through the anticipated workflow of the user. We identified several input components and features the bioGUI program has to include (Fig. A1, in the Appendix).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"bioGUI_approach\">bioGUI approach<\/span><\/h3>\n<p>\u201cThe accessibility of bioinformatics applications is crucial and a challenge of contemporary biology.\u201d<sup id=\"rdp-ebb-cite_ref-MoraisBTW18_4-5\" class=\"reference\"><a href=\"#cite_note-MoraisBTW18-4\">[4]<\/a><\/sup> Particularly, the usage of CLIs poses a problem. Since most bioinformatics applications require the execution of commands on the CL for installation (such as for compilation, adding dependencies to the path variable, etc.), we estimate installation also poses a problem.\n<\/p><p>During the use-case study, and interviews with wet-lab scientists without a computational background (Q. Emslander, 2019, personal communication; L. Jimenez, 2019, personal communication), we found two main problems with bioinformatics applications for scientists, which we want to address with bioGUI: the installation of potentially useful applications and the application's usage. Both problems have in common that they are expected to be performed on the CL. A GUI for achieving the respective tasks in bioinformatics (and beyond) is missing.\n<\/p><p>In particular, the first task of installing bioinformatics applications on a user\u2019s machine poses a few problems. Most bioinformatics applications are written for a Unix operating system, like Linux or Mac OS, while in general Microsoft Windows is the dominant operating system. In order to overcome this problem, bioGUI makes use of WSL on Windows. Even if the user\u2019s OS is already Unix-like, using the CL to install software might prove cumbersome. Thus, in order to support all users, bioGUI uses a cross-platform approach. bioGUI is developed in C++ using the Qt framework.\n<\/p><p>The general workflow for any program using bioGUI is shown in Fig. 1 (above). Given a CL application, the software developer (blue) writes the specific template in an XML-based DSL and can then make this template available, for example, in the bioGUI repository (cyan). Such templates can be automatically retrieved by bioGUI. Upon selection of a template by the user, bioGUI displays the input mask as defined in the template. When the user (yellow) has filled in all parameters, the parameters are collected by bioGUI and assembled into CL arguments, which are used to execute the original CL-only application. Upon completion, simple results (like text-output or images) can be shown in bioGUI directly or within an external application that is opened.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Install_modules\">Install modules<\/span><\/h4>\n<p>Install modules are designed to install applications such that bioGUI can access them. Essentially, install modules are bash scripts which allow an automatic installation of applications into a predefined location. For this purpose, install modules receive several arguments from bioGUI when launched, for example, where to install the application to, the sudo password to fetch packages via a system\u2019s package manager (e.g., aptitude, conda), whether the application should be made available to the user via the system\u2019s PATH variable, etc. Install modules download and install applications and make them available to the user and bioGUI. However, some applications cannot be simply downloaded, but are distributed by installers. For this purpose, the install module template can be extended by further input fields. These must be specified by bioGUI elements, and their values are added to the end of the CL arguments of the install module. An install module can then execute the referenced installer.\n<\/p><p>Finally, an install module should contain the specification of its bioGUI template and could hardcode the path to the installed application. Other constant values, which can already be derived during the installation (e.g., absolute paths to dependencies), could also be defined in the template during this stage.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"bioGUI_templates\">bioGUI templates<\/span><\/h4>\n<p>bioGUI templates are the actual end-user-interface to programs. A bioGUI template defines the look and functions of the UI. Thus it can define how the CL-application is called (with corresponding parameters).\n<\/p><p>Each bioGUI template consists of two parts (Fig. A2). The first part (the <tt><window><\/tt> model) defines the visual appearance of the GUI. The second part (the <tt><execution><\/tt> model) defines the processing logic of the template. Input values from the GUI components are collected and assembled (e.g., pre-\/post-processing steps) to call CL applications. As part of this assembly, input values from the GUI may be transformed using multiple predefined nodes. Concatenations are possible using the <tt><add><\/tt> node, and constant values can be inserted using the <tt><const><\/tt> node. System environment properties, such as the operating system, the computer\u2019s IP address, or specific directories can be collected using the <tt><env><\/tt> node. If the regular nodes are not sufficient, for example, because more complex string manipulations should be made (see use-case study), script nodes may also accept functions written in <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.lua.org\/home.html\" target=\"_blank\">LUA<\/a> or <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ecma-international.org\/ecma-262\/9.0\/\" target=\"_blank\">JavaScript<\/a>.\n<\/p><p>In general, the execution part infers a network with inputs (e.g., GUI elements, other nodes within the execution part) and actions (if, add). For example, the execution network for an application with many sub-commands is exemplarily shown in Fig. A3 (see Appendix).\n<\/p><p>The time to template varies with the application as well as the number of options to be included. A simple template, like the one for MS-EmpiRe<sup id=\"rdp-ebb-cite_ref-AmmarMS-EmpiRe19_22-0\" class=\"reference\"><a href=\"#cite_note-AmmarMS-EmpiRe19-22\">[22]<\/a><\/sup>, can be created within 10 minutes. More comprehensive templates, like the one for HISAT2, usually take about 30 minutes. Time can be saved if only the most important command line options are shown in the GUI. This can be achieved by adding an \u201coptional parameters\u201d input field, where users can insert CL arguments themselves. This is, for instance, shown in the wtdbg2<sup id=\"rdp-ebb-cite_ref-RuanFast19_23-0\" class=\"reference\"><a href=\"#cite_note-RuanFast19-23\">[23]<\/a><\/sup> and spades<sup id=\"rdp-ebb-cite_ref-BankevichSPAdes12_24-0\" class=\"reference\"><a href=\"#cite_note-BankevichSPAdes12-24\">[24]<\/a><\/sup> templates. Adding the install part to a template usually can be done within 15 to 30 minutes, depending on how detailed the build process is documented. The creation of an install module thus takes approximately one hour.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"bioGUI_integration_with_CWL_and_argparse\">bioGUI integration with CWL and argparse<\/span><\/h4>\n<p>The CWL<sup id=\"rdp-ebb-cite_ref-AmstutzCommon16_20-1\" class=\"reference\"><a href=\"#cite_note-AmstutzCommon16-20\">[20]<\/a><\/sup> only describes the CL workflow and neither provides a GUI nor a means to install the desired tool. Due to this more general specification, CWL fits most problems, but specific annotations of inputs, explanations, or the embedding of images is not supported in CWL.\n<\/p><p>While developers can always create templates manually, bioGUI supports developers by offering a template generator from CWL templates or python3 argparse CL parsers. Since there are already many CWL templates available for bioinformatics CL applications, CWL files can be used as a base to automatically generate bioGUI templates from. Using the bioGUI template generator for argparse, it is also possible to automatically generate templates from CWL files (making use of the cwl2argparse program provided by CWL). Our generator takes as input the argparse parser or CWL file and creates input elements for all elements. In case the type of an input is unclear or not supported, the generator falls back to a regular text-input element.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"bioGUI_templates_2\">bioGUI templates<\/span><\/h3>\n<p>Currently more than 25 install modules exist for bioGUI. These represent basically three groups of bioinformatics tasks: next-generation sequencing data analysis and transcriptomics, long read sequencing analysis and assembly, and a more general sequence analysis. In general, these install modules will install the respective application on the local machine. The Circlator<sup id=\"rdp-ebb-cite_ref-HuntCirculator15_25-0\" class=\"reference\"><a href=\"#cite_note-HuntCirculator15-25\">[25]<\/a><\/sup> template allows to pull and use the corresponding Docker image. The available tools, as well as their respective categorization, are listed in Table 1.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 1.<\/b> List of available templates and install modules (starting with Install) for bioGUI; tools marked with \u2713 provide an install module for the operating system of the respective column.\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Module name\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Task\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"2\">Install module\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">WSL and Ubuntu\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Mac OS\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">First Time Mac OS Setup\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Initialization\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2013\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">First Time Ubuntu\/WSL\/apt-get Setup\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Initialization\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2013\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install Ballgown v1.0.1<sup id=\"rdp-ebb-cite_ref-PerteaTranscript16_3-1\" class=\"reference\"><a href=\"#cite_note-PerteaTranscript16-3\">[3]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS transcriptomics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install Bowtie1<sup id=\"rdp-ebb-cite_ref-LangmeadUltra09_26-0\" class=\"reference\"><a href=\"#cite_note-LangmeadUltra09-26\">[26]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install Bowtie2 v2.2.9<sup id=\"rdp-ebb-cite_ref-LangmeadFast12_27-0\" class=\"reference\"><a href=\"#cite_note-LangmeadFast12-27\">[27]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install bwa v0.7.17<sup id=\"rdp-ebb-cite_ref-LiFast09_28-0\" class=\"reference\"><a href=\"#cite_note-LiFast09-28\">[28]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install canu<sup id=\"rdp-ebb-cite_ref-KorenCanu17_29-0\" class=\"reference\"><a href=\"#cite_note-KorenCanu17-29\">[29]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assembly\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install featureCounts<sup id=\"rdp-ebb-cite_ref-LiaoFeature14_30-0\" class=\"reference\"><a href=\"#cite_note-LiaoFeature14-30\">[30]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS transcriptomics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install glimmer302b<sup id=\"rdp-ebb-cite_ref-DelcherIdent07_6-1\" class=\"reference\"><a href=\"#cite_note-DelcherIdent07-6\">[6]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Genome annotation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install graphmap<sup id=\"rdp-ebb-cite_ref-Sovi.C4.87Fast16_31-0\" class=\"reference\"><a href=\"#cite_note-Sovi.C4.87Fast16-31\">[31]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Long read sequencing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install albacore (pip wheel, ONT)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Long read sequencing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install guppy (linux tar.gz, ONT)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Long read sequencing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install hisat2<sup id=\"rdp-ebb-cite_ref-KimGraph19_32-0\" class=\"reference\"><a href=\"#cite_note-KimGraph19-32\">[32]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS transcriptomics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install hmmer-3.1b2<sup id=\"rdp-ebb-cite_ref-WheelerNHMMER13_33-0\" class=\"reference\"><a href=\"#cite_note-WheelerNHMMER13-33\">[33]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sequence analysis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install jellyfish-2.2.6<sup id=\"rdp-ebb-cite_ref-Mar.C3.A7aisAFast11_5-1\" class=\"reference\"><a href=\"#cite_note-Mar.C3.A7aisAFast11-5\">[5]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install minimap2\/miniasm\/racon (gitHub)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assembly (long-read)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install MS-EmpiRe<sup id=\"rdp-ebb-cite_ref-AmmarMS-EmpiRe19_22-1\" class=\"reference\"><a href=\"#cite_note-AmmarMS-EmpiRe19-22\">[22]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS transcriptomics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install PureSeqTM<sup id=\"rdp-ebb-cite_ref-WangPure19_34-0\" class=\"reference\"><a href=\"#cite_note-WangPure19-34\">[34]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sequence analysis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install rMATS-3.2.5<sup id=\"rdp-ebb-cite_ref-ShenRMATS14_35-0\" class=\"reference\"><a href=\"#cite_note-ShenRMATS14-35\">[35]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS transcriptomics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install rnahybrid<sup id=\"rdp-ebb-cite_ref-RehmsmeierFast04_36-0\" class=\"reference\"><a href=\"#cite_note-RehmsmeierFast04-36\">[36]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sequence analysis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install RSEM v1.3.0<sup id=\"rdp-ebb-cite_ref-Li_37-0\" class=\"reference\"><a href=\"#cite_note-Li-37\">[37]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS transcriptomics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install samtools-1.3.1<sup id=\"rdp-ebb-cite_ref-LiTheSeq09_38-0\" class=\"reference\"><a href=\"#cite_note-LiTheSeq09-38\">[38]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install SPAdes v3.13.0<sup id=\"rdp-ebb-cite_ref-BankevichSPAdes12_24-1\" class=\"reference\"><a href=\"#cite_note-BankevichSPAdes12-24\">[24]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assembly (hybrid)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install StringTie v1.3.0<sup id=\"rdp-ebb-cite_ref-PerteaTranscript16_3-2\" class=\"reference\"><a href=\"#cite_note-PerteaTranscript16-3\">[3]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS transcriptomics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install Top Monitor (ssh example)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Technical demo\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install Trimmomatic v0.36<sup id=\"rdp-ebb-cite_ref-BolgerTrimmo14_2-1\" class=\"reference\"><a href=\"#cite_note-BolgerTrimmo14-2\">[2]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install wtdbg2<sup id=\"rdp-ebb-cite_ref-RuanFast19_23-1\" class=\"reference\"><a href=\"#cite_note-RuanFast19-23\">[23]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assembly (long-read)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Install Circulator<sup id=\"rdp-ebb-cite_ref-HuntCirculator15_25-1\" class=\"reference\"><a href=\"#cite_note-HuntCirculator15-25\">[25]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assembly\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Benchmarking_bioGUI_templates\">Benchmarking bioGUI templates<\/span><\/h3>\n<p>Our benchmark comprises of four tasks. The first task is to assemble a bacterial genome from Oxford Nanopore long reads, for which the minimap2<sup id=\"rdp-ebb-cite_ref-LiMini18_39-0\" class=\"reference\"><a href=\"#cite_note-LiMini18-39\">[39]<\/a><\/sup>\/miniasm<sup id=\"rdp-ebb-cite_ref-LiMini16_40-0\" class=\"reference\"><a href=\"#cite_note-LiMini16-40\">[40]<\/a><\/sup>\/racon<sup id=\"rdp-ebb-cite_ref-VaserFast17_41-0\" class=\"reference\"><a href=\"#cite_note-VaserFast17-41\">[41]<\/a><\/sup> pipeline (available as install module from bioGUI) is used. The second task is the quantification of reads from a yeast mRNA sequencing project using Oxford Nanpore Reads and Illumina Reads (EMBL ENA studies PRJNA398797 [MinION] and SAMN00849440 [Illumina]). The quantification is performed using featureCounts from the subread package.<sup id=\"rdp-ebb-cite_ref-LiaoFeature14_30-1\" class=\"reference\"><a href=\"#cite_note-LiaoFeature14-30\">[30]<\/a><\/sup> The third task uses these results to compute differential gene expression. Differential gene expression analysis is performed using MS-EmpiRe in R (install module available<sup id=\"rdp-ebb-cite_ref-AmmarMS-EmpiRe19_22-2\" class=\"reference\"><a href=\"#cite_note-AmmarMS-EmpiRe19-22\">[22]<\/a><\/sup>). Finally the fourth task uses RNAhybrid<sup id=\"rdp-ebb-cite_ref-RehmsmeierFast04_36-1\" class=\"reference\"><a href=\"#cite_note-RehmsmeierFast04-36\">[36]<\/a><\/sup> to predict miRNA binding sites (1,978 murine miRNAs) in 170 sequences of each 200 nt.\n<\/p><p>The results are shown in Table 2. The given run times are wall clock times. The peak RAM consumption has been sampled from the process viewer on the given operating systems (Task Manager on Windows, top on Linux and Mac OS).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"9\"><b>Table 2.<\/b> Benchmarking results for the four selected tasks (see Benchmarking bioGUI templates within the Results section) on the described hardware (see Table A1 in Appendix section). All runs are started via bioGUI.\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Task\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"2\">Linux server\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"2\">Lenovo server\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"2\">Surface Book\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"2\">MacBook Air\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Time\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Peak RAM\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Time\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Peak RAM\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Time\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Peak RAM\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Time\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Peak RAM\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assembly\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10:12 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.8 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23:00 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.5 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30:00 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.5 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">44:30 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.5 GB\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">featureCounts (MinION)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:38 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:54 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">01:12 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">01:30 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 MB\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">featureCounts (Illumina)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">01:13 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">01:41 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">02:22 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">02:30 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25 MB\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DE quantification (MinION)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:19 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.7 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:25 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:28 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:42 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6 GB\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DE quantification (Illumina)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:14 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:19 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:20 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00:31 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4 GB\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">RNAhybrid\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">07:35 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23:00 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13:00 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 MB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16:55 min\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 MB\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>bioGUI is a framework for easy GUI-based usage of CL applications in the life sciences. Using bioGUI, high-quality CL applications can be made accessible to as many researchers as possible. This is achieved by lowering the hurdles to overcome for using bioinformatics applications, particularly on Windows.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Use-case_analysis\">Use-case analysis<\/span><\/h3>\n<p>Our use-case analysis (Appendix section \u201cUse cases\u201d) has revealed several requirements for bioGUI (see the section \u201cMethods\u201d) to enable the user to perform the sequencing analysis and to allow the developer to rapidly create a template (Fig. 2).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Joppich_PeerJ2019_7.jpg\" class=\"image wiki-link\" data-key=\"5b3ef23ba0e849ef972aee99bbdb1266\"><img alt=\"Fig2 Joppich PeerJ2019 7.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/77\/Fig2_Joppich_PeerJ2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> bioGUI use-case study, from a developer\u2019s and user\u2019s perspective, performed on an exemplary RNAseq analysis workflow. The dark-gray underlayed tasks represent the developer\u2019s tasks, and the bright-yellow part represents the analysis pipeline the user wants to execute. Tasks requiring user-action are shown as rectangles and intermediate results are shown in ellipses. Cyan ellipses denote solutions\/results (e.g., template repository) offered by bioGUI. bioGUI starts sub-processes for each task, such that the overhead for any started processes is as small as possible. Upon finishing a task or pipeline, bioGUI can display a notification and open generated output.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>An easy installation (goal 1) is given through the availability of install modules, which can be downloaded from the bioGUI repository and started via the GUI. These also allow additional inputs (e.g., Python wheels for albacore, goal 6).\n<\/p><p>The install modules combine the installation of an application and the creation of the actual GUI template. If the developers employ automatic testing of their software (e.g., build checks with <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/travis-ci.org\/\" target=\"_blank\">Travis<\/a>), the install part resembles a Travis container setup (goal 2): dependencies and the application itself are installed into an operating system. Even if not, most bioinformaticians extensively use Ubuntu and\/or bash scripts. Thus writing a script to install dependencies is not a significantly hard workload. We have reached a seamless and time-efficient creation of templates using an XML-based DSL. XML is particularly helpful as it allows to specify hierarchies and attributes to objects. Using our template generator for CWL and python3-argparse, bioGUI templates can be created even faster (goal 2). The templates are highly flexible in the creation of CL parameters, also due to providing script nodes. By providing install modules and templates, high-quality open-source bioinformatics applications become more accessible to the community.\n<\/p><p>The bioGUI application is cross-platform compatible and only requires a few megabytes of disk space (goal 5). bioGUI implements several possibilities to execute applications (see Fig. A4, in the Appendix). In general, the only runtime overhead involved is the creation of a bash process, which starts the actual program with the assembled CL arguments (goal 5). bioGUI, being a local stand-alone application, has the possibility to target both locally installed and web-based applications, reachable within a controlled environment and with large data. In addition, bioGUI also supports the use of Docker containers, for cases where all other options fail.\n<\/p><p>The UI can be made easily understandable (goal 3). Using text labels, the user can get help on inputs (if specified by the developer), links can be used to provide further information, and, most importantly, tooltips could also hint the user to which information is needed at a certain step.\n<\/p><p>Finally, completed templates can be saved via the \"Save Template\" button in bioGUI, and all available templates can be filtered (goal 4). This enables a user to keep track of performed analyses, and it makes results more reproducible because parameters are saved. Having the possibility to save templates also allows users to easily repeat an analysis with the same parameters. Additionally, using the bioGUI repository, templates can easily be shared among users, making it easier to standardize runs among different users or even institutes.\n<\/p><p>An anonymous survey (with 10 participants) about common problems in using bioinformatics software was also conducted among colleagues (<i>n<\/i> = 4) and undergraduate students or collaborators from the life sciences (<i>n<\/i> = 6, short: collaborators). The results are available in the Appendix section.\n<\/p><p>We asked \u201cWhat were the most cumbersome tasks in accessing and using the software?\u201d, referring to recently used bioinformatics software by the participants. Eight of the 10 participants answered that finding parameters or using the software was cumbersome. This shows that the selected goals for bioGUI address actual problems faced by both experts and regular users. We further asked the participants to install and use graphmap<sup id=\"rdp-ebb-cite_ref-Sovi.C4.87Fast16_31-1\" class=\"reference\"><a href=\"#cite_note-Sovi.C4.87Fast16-31\">[31]<\/a><\/sup>, which was selected because it is reasonably easy to install and use. First the participants were asked to install the tool using the CLI as well as via bioGUI. For this task, all instructions have been provided. The question \u201cHas the installation process been easy?\u201d (0 = No, 5 = Yes) has been answered with an average score of 4.4 for the CLI and 4.8 for bioGUI. Then the participants were asked to align the given reads against a given reference genome, without giving the instructions. Again we asked \u201cHas it been easy to align the reads?\u201d (0 = No, 5 = Yes). Here the CLI scored a 3.5 on average and bioGUI a 4.9. (Fig. A6). This coincidences with the answer to our question \u201cOverall: Which interface was easier to use in your opinion?\u201d (0 = CLI, 5 = GUI). Here the average score was 4. Bioinformaticians and collaborators answered differently: the average bioinformatician was undecided on which interface was easier to use (average 3), but non-bioinformaticians preferred the GUI over the CLI (average 4.5).\n<\/p><p>The survey indicates that there are problems with bioinformatics software regarding installation and usage of CLI tools. These problems can be reduced by providing a GUI for these programs. The more experienced a user is on the CL, the less impact a GUI has. But particularly for non-experts on the CL, a GUI makes it easier to use a program.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"bioGUI_repository\">bioGUI repository<\/span><\/h3>\n<p>We provide a repository of preconfigured templates on our website (see Fig. A5, in the Appendix), where authors and users can search for and browse existing templates or submit new ones. bioGUI can access uploaded templates and save them directly for use. Install modules are provided specifically for WSL and Ubuntu users, which manages dependency resolution and installs applications (locally) onto the user\u2019s device. This currently works in any environment using the aptitude (apt) package manager, but users can submit templates which also support other environments, since install modules are versatile bash scripts. On Mac OS, some install modules support Homebrew for template installation. Install modules download and may severely alter a system (especially if the sudo password is supplied). Thus, submitted install modules are manually curated and are only accessible when no security threat has been identified.\n<\/p><p>The major goal of bioGUI is to enable any scientist to use bioinformatics applications. While we extend the repository on a regular basis depending on our own use, users can also request new templates for applications relevant to them.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Availability_and_extensibility\">Availability and extensibility<\/span><\/h3>\n<p>bioGUI is open-source software, and users or institutions can either use our global bioGUI repository or deploy a custom repository, for example, one which is only reachable within an institution.\n<\/p><p>bioGUI is <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/mjoppich\/bioGUI\" target=\"_blank\">available on GitHub<\/a>. Both source code as well as pre-built binary distributions (for Microsoft Windows, Linux, and Mac OS) are available. While bioGUI will run on any Linux distribution, install modules currently use mainly aptitude as a package manager (e.g., Ubuntu, debian-based distributions). If used on Windows, the same applies for the used WSL-application (Ubuntu 18.04 is recommended). bioGUI has been tested on Microsoft Windows 10, build 17763. On Mac OS, bioGUI uses <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/brew.sh\/\" target=\"_blank\">Homebrew<\/a> to install dependencies. Homebrew does not support a silent, non-interactive installation: the user has to install Homebrew before running the \"First time setup for Mac OS\" install module, which will then install the most common dependencies.\n<\/p><p>While a number of use cases and corresponding components are already included in bioGUI, we encourage users to contribute on GitHub by either pushing their own extensions or opening feature requests. Further documentation (installation and setup guide, how to write templates) is also available via <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/biogui.readthedocs.io\/en\/latest\/index.html\" target=\"_blank\">ReadTheDocs<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Benchmarking_bioGUI_templates_2\">Benchmarking bioGUI templates<\/span><\/h3>\n<p>bioGUI starts a subprocess for each executed program. Thus, the only overhead created by bioGUI itself is the one for running the GUI, which creates less than 1% CPU usage, allocates less than 50 MB, and only performs IO operations when loading a new template (assessed via Sysinternals Process Explorer [Microsoft Sysinternals, 2019]).\n<\/p><p>Nonetheless, we have been interested in demonstrating that many bioinformatics tasks do not require a dedicated server setup but can be performed on regular laptop computers. We thus benchmarked four typical tasks performed using bioGUI.\n<\/p><p>The selected tasks allow a good overview of different demands: Tasks 1 (assembly) and 3 (differential expression analysis) are CPU-bound tasks, while tasks 2 (feature counting) and 4 (miRNA target prediction) are IO-bound. Particularly, Task 2 has a high load of read operations, and Task 4 has a high demand of write operations. We compare these tasks on a dedicated Linux server, one rather powerful Lenovo T470p laptop computer, one Surface Book laptop computer (resource-wise a typical laboratory laptop), and one MacBook Air. The computer specifications are listed in the Table A1 (see Appendix) and results are shown in Table 2 (above). Even though we have not included the alignment of the Illumina yeast reads in this benchmark, it should be noted that this task also runs well on laptop computers. On the Lenovo laptop, the alignment of the SRR453566 sample, consisting of 5,725,730 paired reads, has a peak RAM consumption of 34 MB and took 13:50 minutes, while the Surface Book is even faster at less than eight minutes. This presumably can be explained due to different SSD speeds.\n<\/p><p>The results in Table 2 (above) show that even computationally low-end computers can run bioinformatics tools. More interestingly, these results show that the WSL allows the execution of interesting bioinformatic tools. It can be seen that WSL is slow for IO operations, but it has a comparable speed for in-memory operations. Particularly, tools requiring a lot of IO are considerably faster on the Linux Server (Assembly, RNAhybrid), while the computationally expensive tools like MS-EmpiRe<sup id=\"rdp-ebb-cite_ref-AmmarMS-EmpiRe19_22-3\" class=\"reference\"><a href=\"#cite_note-AmmarMS-EmpiRe19-22\">[22]<\/a><\/sup> and featureCounts<sup id=\"rdp-ebb-cite_ref-LiaoFeature14_30-2\" class=\"reference\"><a href=\"#cite_note-LiaoFeature14-30\">[30]<\/a><\/sup> run within similar times.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The bioGUI framework makes it easy to develop, provide, and use GUIs for CL applications. Particularly for non-computer experts, using CLIs is cumbersome. Providing a GUI and\/or install modules increases accessibility to high-quality bioinformatics applications for these users. bioGUI creates a cross-platform GUI experience for many open-source bioinformatics applications. In particular, bioGUI enables the deployment of academic bioinformatics applications to Microsoft Windows workstations and laptops, as well as Linux or Mac OS.\n<\/p><p>The separation of the GUI components and the program logic allows for the creation of templates in two steps. First, the template developer adds input elements to the window and, second, assembles these inputs according to the needs of the application back into CL arguments. This way almost any CL application can be used with a GUI, enabling many more researchers to use open-source tools. Providing install modules to make Unix applications available to Microsoft Windows users (via WSL) supports this goal.\n<\/p><p>bioGUI can not always replace dedicated GUIs. A tailored UI will still be more usable and user-friendly than any generic solution can be. We experienced this in our use case: certain tasks (e.g., selecting options) require special solutions, let alone from displaying or interpreting the results. However, especially with the install module concept, we aim to provide a seamless installation and create the possibility to run CL applications by all scientists. Using the bioGUI framework, simple GUIs can be constructed. But these simple GUIs already help to make bioinformatic tools more accessible by making execution and usage of these tools more comfortable.\n<\/p><p>Using bioGUI, it becomes a simple exercise to use supported CL applications from a GUI. Currently, there are already installed modules and templates for more than 25 applications in our bioGUI repository. bioGUI lowers the burden to use excellent applications, allowing more scientists a better analysis of their data. With bioGUI, it is not necessary to understand how to use and navigate on the CL; instead, the focus is set on the applications, its method and parameters, and finally the data.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Appendix\">Appendix<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Use_cases\">Use cases<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Non-computer_expert\">Non-computer expert<\/span><\/h4>\n<p>Many researchers work in small labs without any significant IT support. The computers in their labs mostly run Microsoft Windows, and PhD students often have to bring their own devices (because the institute does not provide such working devices). Particularly in the life sciences, users can profit greatly from existing open-source software. However, installing major bioinformatics applications on such lab computers often poses a problem: administrators (if existent) have little time to deploy new applications, or there is no support in installing new applications at all. If the users are not computer experts, installing and using command line (CL) tools may be cumbersome for them (see the user surveys in this appendix). While there are users that can use the CL efficiently, the cited literature and our personal experience shows that there are many users who do not feel comfortable on the CL. This does not mean that they don\u2019t want to learn it or are incapable of learning it, but their focus simply does not lie in learning to use the CL. Instead, they want to get the results for their data fast, reliably, and without issue. One of these users is Luisa.\n<\/p><p>Oxford Nanopore Sequencing is becoming more and more popular, and even the sequencing hardware can be found in more and more biological laboratories, like in Luisa\u2019s. Particularly important for MinION sequencing is the post-processing of the actual raw read data. While in previous versions base-calling was directly performed in the cloud by Oxford Nanopore, this has now been pushed back to the client side. Thus, despite having the sequencing data on her laptop, Luisa must still retrieve the sequences herself, using, for instance, the Albacore basecaller (if they don\u2019t want to rely on LiveBasecalling). Unfortunately, like many bioinformatics packages, the basecaller only comes as a python CL program. Additionally, the download is only available as a Python wheel, which means there is no UI-based setup available. Luisa thus needs assistance for the installation of the Python wheel, as well as starting the basecalling process. After the reads are basecalled, reads need to be aligned to a reference genome. While reference genomes in a correct format exist on her lab computer (or can easily be downloaded from the web), the CL program to map the reads is available only from GitHub to be installed from source. Luisa has trouble using the CL to clone the repository, compile, and use the CL application.\n<\/p><p>Luisa does not require a custom analysis of her data but rather wants to initially screen her data in a simple, basic, and robust analysis. She is mostly busy in her lab, hence an analysis has to be prepared fast, and parameters should be stored for later reference. For this, a local searchable database of saved templates is needed.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Software_developers\">Software developers<\/span><\/h4>\n<p>A developer finished his sequence alignment program. The project is already published on GitHub and in a journal, but only a few people start using it. From the issues and feature requests on GitHub it can be seen that mainly other bioinformaticians use the program. Thus, the developer decides that the program should be accessible to more researchers and looks for ways to make the program usable by everyone. Since the program is written in Python, it is cross-platform compatible. However, it is noticed that domain experts do not install and use the program. Thus the developer must look for an easy way to distribute the application and make it accessible to more researchers. The developer\u2019s time is limited, having other projects waiting. There is also little support for developing a GUI from colleagues, as they have different views on the extent of autonomy a wet lab researcher should have regarding sequencing analysis.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"bioGUI_paper_mockup\">bioGUI paper mockup<\/span><\/h3>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:FigA1_Joppich_PeerJ2019_7.jpg\" class=\"image wiki-link\" data-key=\"4c07dfa10a10d34246e44cd6435c9f95\"><img alt=\"FigA1 Joppich PeerJ2019 7.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/18\/FigA1_Joppich_PeerJ2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure A1.<\/b> bioGUI mockup showing the elements a template could be made of. The GUI has a searchable list of installed templates as well as a link to our repository of templates. The right side is reserved to the currently displayed GUI template. Here a structured view of the available parameters, as well as hints for filling these, is shown to the user. Finally, the user has the possibility to run the program by clicking a button and to see the program\u2019s output.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Extending_templates_with_script_nodes\">Extending templates with script nodes<\/span><\/h3>\n<p>Often it is required to perform string manipulations (e.g., remove file extensions) for CL arguments. For instance, the example below takes as input a HISAT2 index file and removes the file extension, such that the index will be accepted by HISAT2. For evaluation of this node, the <tt>evaluate<\/tt> function is called with the <tt>argv<\/tt> references as input parameters. The last return value of the script\u2019s call stack is taken as the output value of the script -node.\n<\/p><p><b>Listing 1.<\/b> bioGUI script node with LUA function example. Upon evaluating this node, the evaluate function will be called with the arguments listed in the <tt>argv<\/tt> attribute of the script node.\n<\/p><p><code><script id=\u201chisat_index_rel\u201d argv=\u201c${hisat_index_rel_raw}\u201d>\n<\/p><p><![CDATA[\n<\/p><p>function evaluate(arg1)\n<\/p><p>\u2003\u2003\u2003\u2003if (string.match(arg1, \u201c.%d.ht2$\u201d)) then\n<\/p><p>\u2003\u2003\u2003\u2003\u2003\u2003return(string.sub(arg1, 0, arg1:find(\u201c.%d.ht2$\u201d)-1))\n<\/p><p>\u2003\u2003\u2003\u2003end\n<\/p><p>\u2003\u2003\u2003return(arg1)\n<\/p><p>\u2003\u2003end\n<\/p><p>]]>\n<\/p><p><\/script><\/code>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Evaluating_a_bioGUI_template\">Evaluating a bioGUI template<\/span><\/h3>\n<p>In Fig. A2, the process of assembling a CL call from the shown bioGUI template is explained. First, the creation of the <tt><window><\/tt> model (dark gray) will be explained, followed by the creation of the CL arguments using the <tt><execution><\/tt> model (shaded).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:FigA2_Joppich_PeerJ2019_7.jpg\" class=\"image wiki-link\" data-key=\"673c5aeb8e2fbbb3fddb4a141dbc4d11\"><img alt=\"FigA2 Joppich PeerJ2019 7.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3d\/FigA2_Joppich_PeerJ2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure A2.<\/b> Template construction and evaluation in bioGUI. First, the dark gray window part is evaluated to create the GUI. Once the user clicks the run button, the execution part of the template (shaded) is executed by constructing and starting the assembled system call. This system call is constructed in three steps by replacing variables with evaluated terms from the user\u2019s input. Blue lines indicate the visual element a returned value (cyan lines) is taken from. Helper\/intermediate nodes to be evaluated are shown in light yellow.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The window component consists of four different components, which are grouped in a vertical layout (default for window component). A label describing the input file dialog is placed on the main window, followed by the actual file dialog with ID input. Then a group box with title and a checkable status is created, which contains an output file dialog. Finally, the action button, which starts the CL call assembly and the subprocess of this program, and the text output elements are created.\n<\/p><p>When the user has entered all desired data and clicks the action button, the execution phase defined by the execution model will be launched. Therefore the program defined in the execute element is started. For this, the parameters (param) must be assembled. Any text within <tt>${var}<\/tt> is interpreted as a reference to a variable \"var\" or the value of a GUI element with id \"var.\" Thus, the CL is successively assembled. At first the <tt>${input}<\/tt> element is interpreted and retrieves the value from the input file dialog as this element matches the id. Next the <tt>${output}<\/tt> is interpreted. The <tt>${output}<\/tt> refers to an \"if\" construct in the execution part, which compares the value of the element with id \"os\" to the string \"TRUE\" (which indicated whether the groupbox is checked). If this value is true, this node evaluates to <tt>netcat 192.168.1.100 55025<\/tt>, otherwise to <tt>tee -a {output file path}<\/tt>. Finally, the program \"sh\" is executed with the created CL arguments. For instance, if the group box is checked, the <tt>sh -c \u201ccat inputFile | netcat 192.168.1.100 55025\u201d<\/tt> will be executed. A full reference of all input types as well as all execution nodes is available online.\n<\/p><p>In fact, the evaluation of the execution network resembles the simulation of a petri net (Fig. A3). Each node in the execution network is a place, and its modification\/function is the transition, which requires values for all its input places, to generate the output token.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:FigA3_Joppich_PeerJ2019_7.jpg\" class=\"image wiki-link\" data-key=\"fb08ba1550d6e10558ff8495cd9e4ae4\"><img alt=\"FigA3 Joppich PeerJ2019 7.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a9\/FigA3_Joppich_PeerJ2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure A3.<\/b> (<b>A<\/b>) An automatically generated bioGUI template from the poreSTAT (Internal tool for minION sequencing analysis) python argument parser. (<b>B<\/b>) The resulting execution network for the bioGUI template shown in (A). The central node represents the fully assembled CL argument (yellow).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Running_programs_via_bioGUI\">Running programs via bioGUI<\/span><\/h3>\n<p>Program execution via bioGUI can be accomplished via different paths, which are shown in Fig. A4. The easiest way is to execute a native program (one that runs natively on the operating system, e.g., Docker). Then all output can be piped to bioGUI to display this to the user. If the host is a Microsoft Windows 10 OS, bioGUI can also run Unix programs via WSL. Then the Unix program runs natively in a WSL bash. The resulting output can be transferred to bioGUI via pipes. Of course, for both native and WSL processes, the output can also be transferred via netcat to bioGUI. The transfer of the GUI template within install modules is an example. If a process runs on a remote computer, the output can be transferred to bioGUI also via network, for example, netcat. Such a process can, for instance, be started by calling ssh from bioGUI with appropriate parameters. Finally bioGUI can also send HTTP POST requests to web services and accepts an HTTP response as answer. This output can also be displayed by bioGUI.\n<\/p><p>Since the Docker engine is a local, native process, bioGUI also supports the use of Docker containers. The Circlator template is an example of how this can be implemented.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:FigA4_Joppich_PeerJ2019_7.jpg\" class=\"image wiki-link\" data-key=\"24f23bf7bde1d717fa6e88740a7eb1e7\"><img alt=\"FigA4 Joppich PeerJ2019 7.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/16\/FigA4_Joppich_PeerJ2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure A4.<\/b> Possibilities for running bioGUI: locally via processes, on a network via ssh or on the web via HTTP request\/response. Straight arrow (purple): HTTP execution mode; Dotted arrow (green): Docker execution; Dotdashed arrow(orange): bash\/WSL execution; Dashed arrow(cyan): remote\/ssh execution.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Hardware_specification_for_benchmarks\">Hardware specification for benchmarks<\/span><\/h3>\n<p>The relevant hardware for benchmarking bioGUI is summarized in Table A1.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table A1.<\/b>Hardware used to benchmark bioGUI\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Computer name\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPU\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RAM\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Storage\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Linux server\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Intel Xeon W-2145 CPU @ 3.70 GHz<br \/>8 cores (+8 HT cores)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">128 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Samsung SSD 860, 1 TB SSD\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lenovo laptop (T470p)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Intel Core i7-7820HQ @ 2.9 GHz<br \/>4 cores (+4 HT cores)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">32 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Samsung MZVLB1T0HALR, 1 TB SSD\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Microsoft surface book\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Intel Core i5-6300U @ 2.4 GHz<br \/>2 cores (+2 HT cores)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Samsung MZFLV128HCGR, 128 GB SSD\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Apple MacBook Air (mid 2012)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Intel Core i5 @ 1.7 GHz\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8 GB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">128 GB SSD\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Template_access\">Template access<\/span><\/h3>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:FigA5_Joppich_PeerJ2019_7.jpg\" class=\"image wiki-link\" data-key=\"2ac11e1d81f0f2997c2699f57dcc815b\"><img alt=\"FigA5 Joppich PeerJ2019 7.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/7d\/FigA5_Joppich_PeerJ2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure A5.<\/b> (<b>A<\/b>) On our website a list of already existing templates can be browsed. Besides the description and author, also the type (install module or template) is shown. (<b>B<\/b>) All uploaded templates can be downloaded directly from within bioGUI. bioGUI allows to search in\/filter all available install modules and templates.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"User_survey\">User survey<\/span><\/h3>\n<p>A user survey with 10 participants (four bioinformaticians, and six collaborators consisting of two undergraduate bioinformatics students and four external collaborators) was performed. The derived results are shown in Table A2, and the raw data are shown in Table A3.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table A2.<\/b> Derived user survey results from the given answers\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\"><i>n<\/i>\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Median\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Mean\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\"><i>p<\/i>-value\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Variance\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Better interface bio\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Better interface collab\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Better interface all\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.75\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Easy to align CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.833\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Easy to align bioGUI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.90\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.0098\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Easy to install CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.711\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Easy to install bioGUI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.80\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2023\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.178\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"11\"><b>Table A3.<\/b> Relevant participant answers for the performed user survey and the results in Table A2\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 1\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 2\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 3\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 4\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 5\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 6\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 7\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 8\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 9\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant 10\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Usertype (0 = bioinformatician, 1 = student, 2 = collaborator)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Which kind of user-interface does the tool have?\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GUI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GUI\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">What were the most cumbersome tasks in accessing and using the software?\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dependencies\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Using software, finding settings\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Finding settings, options needed\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Installing software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Finding settings, options needed\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dependencies, starting the software, using the software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Finding settings, options needed\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Finding settings, options needed\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Using the software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Finding settings, options needed\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI: Has the installation process been easy? (0 = NO, 5 = YES)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CLI: Has it been easy to align the reads? (0 = NO, 5 = YES)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">bioGUI: Has the installation process been easy? (0 = NO, 5 = YES)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">bioGUI: Has it been easy to align the reads? (0 = NO, 5 = YES)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Overall: Which interface was easier to use in your opinion? (0 = CLI, 5 = GUI)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:FigA6_Joppich_PeerJ2019_7.jpg\" class=\"image wiki-link\" data-key=\"62794b6a0c3027f73f7c960e63ff9c57\"><img alt=\"FigA6 Joppich PeerJ2019 7.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/ff\/FigA6_Joppich_PeerJ2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure A6.<\/b> Scores given by the 10 participants on the question \u201cHas it been easy to align the reads?\u201d after performing the task using the CLI and bioGUI. These results show that most participants found the task easier using bioGUI, but for no-one it was harder to use bioGUI.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Supplemental_information\">Supplemental information<\/span><\/h2>\n<ul><li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dfzljdn9uc3pi.cloudfront.net\/2019\/8111\/1\/survey.pdf\" target=\"_blank\">DOI 10.7717\/peerj.8111\/supp-1<\/a> - Survey questions on command-line tools and bioGUI: This is the original survey used to assess problems with current bioinformatics applications. (PDF)<\/li><\/ul>\n<ul><li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dfzljdn9uc3pi.cloudfront.net\/2019\/8111\/1\/survey_with_diagram.xlsx\" target=\"_blank\">DOI 10.7717\/peerj.8111\/supp-2<\/a> - Answers on the survey on command-line tools and bioGUI: Each column represents a single participant. Questions are in rows. (XLXS)<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We thank Luisa F. Jimenez-Soto and Gergely Csaba for their valuable input as well as for reviewing the manuscript. We thank the participants in our survey for their time. We thank the reviewers for their constructive feedback.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h3>\n<p>Markus Joppich conceived and designed the experiments, performed the experiments, analyzed the data, prepared figures and\/or tables, authored or reviewed drafts of the paper, and approved the final draft. Ralf Zimmer conceived and designed the experiments, analyzed the data, prepared figures and\/or tables, authored or reviewed drafts of the paper, and approved the final draft.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work was supported by the Deutsche Forschungsgemeinschaft (Collaborative Research Centre SFB 1123-2\/Z2). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_availability\">Data availability<\/span><\/h3>\n<p>The bioGUI documentation is available <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/biogui.readthedocs.io\/en\/latest\/\" target=\"_blank\">here<\/a>. In order to set up Windows Subsystem for Linux (required for using bioGUI on Windows), follow the steps documented <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/biogui.readthedocs.io\/en\/latest\/build_wsl.html\" target=\"_blank\">here<\/a>. bioGUI is open-source software. Releases and code are available on the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/mjoppich\/bioGUI\" target=\"_blank\">GitHub project page<\/a>. Additional software (cwl2biogui) is available <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/mjoppich\/bioGUItools\" target=\"_blank\">here<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-GrabherrFull11-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GrabherrFull11_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Grabherr, M.G.; Haas, B.J.; Yassour, M. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3571712\" target=\"_blank\">\"Full-length transcriptome assembly from RNA-Seq data without a reference genome\"<\/a>. <i>Nature Biotechnology<\/i> <b>29<\/b> (7): 644\u201352. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt.1883\" target=\"_blank\">10.1038\/nbt.1883<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3571712\/\" target=\"_blank\">PMC3571712<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21572440\" target=\"_blank\">21572440<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3571712\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3571712<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Full-length+transcriptome+assembly+from+RNA-Seq+data+without+a+reference+genome&rft.jtitle=Nature+Biotechnology&rft.aulast=Grabherr%2C+M.G.%3B+Haas%2C+B.J.%3B+Yassour%2C+M.+et+al.&rft.au=Grabherr%2C+M.G.%3B+Haas%2C+B.J.%3B+Yassour%2C+M.+et+al.&rft.date=2011&rft.volume=29&rft.issue=7&rft.pages=644%E2%80%9352&rft_id=info:doi\/10.1038%2Fnbt.1883&rft_id=info:pmc\/PMC3571712&rft_id=info:pmid\/21572440&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3571712&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BolgerTrimmo14-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BolgerTrimmo14_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-BolgerTrimmo14_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bolger, A.M.; Lohse, M.; Usadel, B. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4103590\" target=\"_blank\">\"Trimmomatic: A flexible trimmer for Illumina sequence data\"<\/a>. <i>Bioinformatics<\/i> <b>30<\/b> (15): 2114-20. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtu170\" target=\"_blank\">10.1093\/bioinformatics\/btu170<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4103590\/\" target=\"_blank\">PMC4103590<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24695404\" target=\"_blank\">24695404<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4103590\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4103590<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Trimmomatic%3A+A+flexible+trimmer+for+Illumina+sequence+data&rft.jtitle=Bioinformatics&rft.aulast=Bolger%2C+A.M.%3B+Lohse%2C+M.%3B+Usadel%2C+B.&rft.au=Bolger%2C+A.M.%3B+Lohse%2C+M.%3B+Usadel%2C+B.&rft.date=2014&rft.volume=30&rft.issue=15&rft.pages=2114-20&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtu170&rft_id=info:pmc\/PMC4103590&rft_id=info:pmid\/24695404&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4103590&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PerteaTranscript16-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PerteaTranscript16_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-PerteaTranscript16_3-1\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-PerteaTranscript16_3-2\">3.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pertea, M.; Kim, D.; Pertea, G.M. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5032908\" target=\"_blank\"><i>Transcript-level expression analysis of RNA-seq experiments with HISAT, StringTie and Ballgown<\/i><\/a>. <b>11<\/b>. pp. 1650\u201367. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnprot.2016.095\" target=\"_blank\">10.1038\/nprot.2016.095<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5032908\/\" target=\"_blank\">PMC5032908<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27560171\" target=\"_blank\">27560171<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5032908\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5032908<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Transcript-level+expression+analysis+of+RNA-seq+experiments+with+HISAT%2C+StringTie+and+Ballgown&rft.aulast=Pertea%2C+M.%3B+Kim%2C+D.%3B+Pertea%2C+G.M.+et+al.&rft.au=Pertea%2C+M.%3B+Kim%2C+D.%3B+Pertea%2C+G.M.+et+al.&rft.date=2016&rft.volume=11&rft.issue=9&rft.pages=pp.+1650%E2%80%9367&rft_id=info:doi\/10.1038%2Fnprot.2016.095&rft_id=info:pmc\/PMC5032908&rft_id=info:pmid\/27560171&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5032908&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoraisBTW18-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MoraisBTW18_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-MoraisBTW18_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-MoraisBTW18_4-2\">4.2<\/a><\/sup> <sup><a href=\"#cite_ref-MoraisBTW18_4-3\">4.3<\/a><\/sup> <sup><a href=\"#cite_ref-MoraisBTW18_4-4\">4.4<\/a><\/sup> <sup><a href=\"#cite_ref-MoraisBTW18_4-5\">4.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Morais, D.; Roesch, L.F.W.; Redmile-Gordon, M. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6074753\" target=\"_blank\"><i>BTW-Bioinformatics Through Windows: An easy-to-install package to analyze marker gene data<\/i><\/a>. <b>6<\/b>. pp. e5299. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717%2Fpeerj.5299\" target=\"_blank\">10.7717\/peerj.5299<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6074753\/\" target=\"_blank\">PMC6074753<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30083449\" target=\"_blank\">30083449<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6074753\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6074753<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=BTW-Bioinformatics+Through+Windows%3A+An+easy-to-install+package+to+analyze+marker+gene+data&rft.aulast=Morais%2C+D.%3B+Roesch%2C+L.F.W.%3B+Redmile-Gordon%2C+M.+et+al.&rft.au=Morais%2C+D.%3B+Roesch%2C+L.F.W.%3B+Redmile-Gordon%2C+M.+et+al.&rft.date=2018&rft.volume=6&rft.pages=pp.+e5299&rft_id=info:doi\/10.7717%2Fpeerj.5299&rft_id=info:pmc\/PMC6074753&rft_id=info:pmid\/30083449&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6074753&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mar.C3.A7aisAFast11-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Mar.C3.A7aisAFast11_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-Mar.C3.A7aisAFast11_5-1\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mar\u00e7ais, G.; Kingsford, C. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3051319\" target=\"_blank\">\"A fast, lock-free approach for efficient parallel counting of occurrences of k-mers\"<\/a>. <i>Bioinformatics<\/i> <b>27<\/b> (6): 764\u201370. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtr011\" target=\"_blank\">10.1093\/bioinformatics\/btr011<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3051319\/\" target=\"_blank\">PMC3051319<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21217122\" target=\"_blank\">21217122<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3051319\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3051319<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+fast%2C+lock-free+approach+for+efficient+parallel+counting+of+occurrences+of+k-mers&rft.jtitle=Bioinformatics&rft.aulast=Mar%C3%A7ais%2C+G.%3B+Kingsford%2C+C.&rft.au=Mar%C3%A7ais%2C+G.%3B+Kingsford%2C+C.&rft.date=2011&rft.volume=27&rft.issue=6&rft.pages=764%E2%80%9370&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtr011&rft_id=info:pmc\/PMC3051319&rft_id=info:pmid\/21217122&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3051319&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DelcherIdent07-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DelcherIdent07_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-DelcherIdent07_6-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Delcher, A.L.; Bratke, K.A.; Powers, E.C. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2387122\" target=\"_blank\">\"Identifying bacterial genes and endosymbiont DNA with Glimmer\"<\/a>. <i>Bioinformatics<\/i> <b>23<\/b> (6): 673\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtm009\" target=\"_blank\">10.1093\/bioinformatics\/btm009<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2387122\/\" target=\"_blank\">PMC2387122<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17237039\" target=\"_blank\">17237039<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2387122\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2387122<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identifying+bacterial+genes+and+endosymbiont+DNA+with+Glimmer&rft.jtitle=Bioinformatics&rft.aulast=Delcher%2C+A.L.%3B+Bratke%2C+K.A.%3B+Powers%2C+E.C.+et+al.&rft.au=Delcher%2C+A.L.%3B+Bratke%2C+K.A.%3B+Powers%2C+E.C.+et+al.&rft.date=2007&rft.volume=23&rft.issue=6&rft.pages=673%E2%80%939&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtm009&rft_id=info:pmc\/PMC2387122&rft_id=info:pmid\/17237039&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2387122&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HosnyAlgoRun16-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HosnyAlgoRun16_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hosny, A.; Vera-Licona, P.; Laubenbacher, R. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6280798\" target=\"_blank\">\"AlgoRun: A Docker-based packaging system for platform-agnostic implemented algorithms\"<\/a>. <i>Bioinformatics<\/i> <b>32<\/b> (15): 2396\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtw120\" target=\"_blank\">10.1093\/bioinformatics\/btw120<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6280798\/\" target=\"_blank\">PMC6280798<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27153722\" target=\"_blank\">27153722<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6280798\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6280798<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=AlgoRun%3A+A+Docker-based+packaging+system+for+platform-agnostic+implemented+algorithms&rft.jtitle=Bioinformatics&rft.aulast=Hosny%2C+A.%3B+Vera-Licona%2C+P.%3B+Laubenbacher%2C+R.+et+al.&rft.au=Hosny%2C+A.%3B+Vera-Licona%2C+P.%3B+Laubenbacher%2C+R.+et+al.&rft.date=2016&rft.volume=32&rft.issue=15&rft.pages=2396%E2%80%938&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtw120&rft_id=info:pmc\/PMC6280798&rft_id=info:pmid\/27153722&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6280798&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchadtTheChang12-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchadtTheChang12_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schadt, E.E. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3472686\" target=\"_blank\">\"The changing privacy landscape in the era of big data\"<\/a>. <i>Molecular Systems Biology<\/i> <b>8<\/b>: 612. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fmsb.2012.47\" target=\"_blank\">10.1038\/msb.2012.47<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3472686\/\" target=\"_blank\">PMC3472686<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22968446\" target=\"_blank\">22968446<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3472686\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3472686<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+changing+privacy+landscape+in+the+era+of+big+data&rft.jtitle=Molecular+Systems+Biology&rft.aulast=Schadt%2C+E.E.&rft.au=Schadt%2C+E.E.&rft.date=2012&rft.volume=8&rft.pages=612&rft_id=info:doi\/10.1038%2Fmsb.2012.47&rft_id=info:pmc\/PMC3472686&rft_id=info:pmid\/22968446&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3472686&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GymrekIdent13-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GymrekIdent13_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gymrek, M.; McGuire, A.L.; Golan, D. et al. (2013). \"Identifying personal genomes by surname inference\". <i>Science<\/i> <b>339<\/b> (6117): 321\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.1229566\" target=\"_blank\">10.1126\/science.1229566<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23329047\" target=\"_blank\">23329047<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identifying+personal+genomes+by+surname+inference&rft.jtitle=Science&rft.aulast=Gymrek%2C+M.%3B+McGuire%2C+A.L.%3B+Golan%2C+D.+et+al.&rft.au=Gymrek%2C+M.%3B+McGuire%2C+A.L.%3B+Golan%2C+D.+et+al.&rft.date=2013&rft.volume=339&rft.issue=6117&rft.pages=321%E2%80%934&rft_id=info:doi\/10.1126%2Fscience.1229566&rft_id=info:pmid\/23329047&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlberBiostar16-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlberBiostar16_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Albert, I. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/biostar.myshopify.com\/\" target=\"_blank\">\"The Biostar Handbook\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/biostar.myshopify.com\/\" target=\"_blank\">https:\/\/biostar.myshopify.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Biostar+Handbook&rft.atitle=&rft.aulast=Albert%2C+I.&rft.au=Albert%2C+I.&rft.date=2016&rft_id=https%3A%2F%2Fbiostar.myshopify.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PavelinBioinfo12-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PavelinBioinfo12_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pavelin, K.; Cham, J.A.; de Matos, P. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3395592\" target=\"_blank\">\"Bioinformatics meets user-centred design: a perspective\"<\/a>. <i>PLoS Computational Biology<\/i> <b>8<\/b> (7): e1002554. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1002554\" target=\"_blank\">10.1371\/journal.pcbi.1002554<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3395592\/\" target=\"_blank\">PMC3395592<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22807660\" target=\"_blank\">22807660<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3395592\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3395592<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+meets+user-centred+design%3A+a+perspective&rft.jtitle=PLoS+Computational+Biology&rft.aulast=Pavelin%2C+K.%3B+Cham%2C+J.A.%3B+de+Matos%2C+P.+et+al.&rft.au=Pavelin%2C+K.%3B+Cham%2C+J.A.%3B+de+Matos%2C+P.+et+al.&rft.date=2012&rft.volume=8&rft.issue=7&rft.pages=e1002554&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1002554&rft_id=info:pmc\/PMC3395592&rft_id=info:pmid\/22807660&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3395592&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmithTheBattle13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmithTheBattle13_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smith, D.R. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3778374\" target=\"_blank\">\"The battle for user-friendly bioinformatics\"<\/a>. <i>Frontiers in Genetics<\/i> <b>4<\/b>: 187. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffgene.2013.00187\" target=\"_blank\">10.3389\/fgene.2013.00187<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3778374\/\" target=\"_blank\">PMC3778374<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24065986\" target=\"_blank\">24065986<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3778374\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3778374<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+battle+for+user-friendly+bioinformatics&rft.jtitle=Frontiers+in+Genetics&rft.aulast=Smith%2C+D.R.&rft.au=Smith%2C+D.R.&rft.date=2013&rft.volume=4&rft.pages=187&rft_id=info:doi\/10.3389%2Ffgene.2013.00187&rft_id=info:pmc\/PMC3778374&rft_id=info:pmid\/24065986&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3778374&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmithBuying15-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmithBuying15_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smith, D.R. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4501248\" target=\"_blank\"><i>Buying in to bioinformatics: An introduction to commercial sequence analysis software<\/i><\/a>. <b>16<\/b>. p. 700-9. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbu030\" target=\"_blank\">10.1093\/bib\/bbu030<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4501248\/\" target=\"_blank\">PMC4501248<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25183247\" target=\"_blank\">25183247<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4501248\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4501248<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Buying+in+to+bioinformatics%3A+An+introduction+to+commercial+sequence+analysis+software&rft.aulast=Smith%2C+D.R.&rft.au=Smith%2C+D.R.&rft.date=2015&rft.volume=16&rft.issue=4&rft.pages=p.+700-9&rft_id=info:doi\/10.1093%2Fbib%2Fbbu030&rft_id=info:pmc\/PMC4501248&rft_id=info:pmid\/25183247&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4501248&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VisneRGG09-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VisneRGG09_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-VisneRGG09_14-1\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-VisneRGG09_14-2\">14.2<\/a><\/sup> <sup><a href=\"#cite_ref-VisneRGG09_14-3\">14.3<\/a><\/sup> <sup><a href=\"#cite_ref-VisneRGG09_14-4\">14.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Visne, I.; Dilaveroglu, E.; Vierlinger, K. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2653488\" target=\"_blank\"><i>RGG: A general GUI Framework for R scripts<\/i><\/a>. <b>10<\/b>. p. 74. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-10-74\" target=\"_blank\">10.1186\/1471-2105-10-74<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2653488\/\" target=\"_blank\">PMC2653488<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19254356\" target=\"_blank\">19254356<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2653488\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2653488<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=RGG%3A+A+general+GUI+Framework+for+R+scripts&rft.aulast=Visne%2C+I.%3B+Dilaveroglu%2C+E.%3B+Vierlinger%2C+K.+et+al.&rft.au=Visne%2C+I.%3B+Dilaveroglu%2C+E.%3B+Vierlinger%2C+K.+et+al.&rft.date=2009&rft.volume=10&rft.pages=p.+74&rft_id=info:doi\/10.1186%2F1471-2105-10-74&rft_id=info:pmc\/PMC2653488&rft_id=info:pmid\/19254356&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2653488&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AfganTheGal16-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AfganTheGal16_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-AfganTheGal16_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Afgan, E.; Baker, D.; van den Beek, M. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987906\" target=\"_blank\">\"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update\"<\/a>. <i>Nucleic Acids Research<\/i> <b>44<\/b> (W1): W3\u2013W10. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkw343\" target=\"_blank\">10.1093\/nar\/gkw343<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4987906\/\" target=\"_blank\">PMC4987906<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27137889\" target=\"_blank\">27137889<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987906\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987906<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Galaxy+platform+for+accessible%2C+reproducible+and+collaborative+biomedical+analyses%3A+2016+update&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Afgan%2C+E.%3B+Baker%2C+D.%3B+van+den+Beek%2C+M.+et+al.&rft.au=Afgan%2C+E.%3B+Baker%2C+D.%3B+van+den+Beek%2C+M.+et+al.&rft.date=2016&rft.volume=44&rft.issue=W1&rft.pages=W3%E2%80%93W10&rft_id=info:doi\/10.1093%2Fnar%2Fgkw343&rft_id=info:pmc\/PMC4987906&rft_id=info:pmid\/27137889&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4987906&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HunterYabi12-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HunterYabi12_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hunter, A.A.; Macgregor, A.B.; Szabo, T.O. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3298538\" target=\"_blank\">\"Yabi: An online research environment for grid, high performance and cloud computing\"<\/a>. <i>Source Code for Biology and Medicine<\/i> <b>7<\/b> (1): 1. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1751-0473-7-1\" target=\"_blank\">10.1186\/1751-0473-7-1<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3298538\/\" target=\"_blank\">PMC3298538<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22333270\" target=\"_blank\">22333270<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3298538\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3298538<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Yabi%3A+An+online+research+environment+for+grid%2C+high+performance+and+cloud+computing&rft.jtitle=Source+Code+for+Biology+and+Medicine&rft.aulast=Hunter%2C+A.A.%3B+Macgregor%2C+A.B.%3B+Szabo%2C+T.O.+et+al.&rft.au=Hunter%2C+A.A.%3B+Macgregor%2C+A.B.%3B+Szabo%2C+T.O.+et+al.&rft.date=2012&rft.volume=7&rft.issue=1&rft.pages=1&rft_id=info:doi\/10.1186%2F1751-0473-7-1&rft_id=info:pmc\/PMC3298538&rft_id=info:pmid\/22333270&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3298538&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XuRNA14-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-XuRNA14_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-XuRNA14_17-1\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-XuRNA14_17-2\">17.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xu, G.; Strong, M.J.; Lacey, M.R. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3934900\" target=\"_blank\">\"RNA CoMPASS: A dual approach for pathogen and host transcriptome analysis of RNA-seq datasets\"<\/a>. <i>PLoS One<\/i> <b>9<\/b> (2): e89445. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0089445\" target=\"_blank\">10.1371\/journal.pone.0089445<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3934900\/\" target=\"_blank\">PMC3934900<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24586784\" target=\"_blank\">24586784<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3934900\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3934900<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=RNA+CoMPASS%3A+A+dual+approach+for+pathogen+and+host+transcriptome+analysis+of+RNA-seq+datasets&rft.jtitle=PLoS+One&rft.aulast=Xu%2C+G.%3B+Strong%2C+M.J.%3B+Lacey%2C+M.R.+et+al.&rft.au=Xu%2C+G.%3B+Strong%2C+M.J.%3B+Lacey%2C+M.R.+et+al.&rft.date=2014&rft.volume=9&rft.issue=2&rft.pages=e89445&rft_id=info:doi\/10.1371%2Fjournal.pone.0089445&rft_id=info:pmc\/PMC3934900&rft_id=info:pmid\/24586784&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3934900&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AnslanPipeCraft17-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AnslanPipeCraft17_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-AnslanPipeCraft17_18-1\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Anslan, S.; Bahram, M.; Hiirsalu, I. et al. (2017). \"PipeCraft: Flexible open-source toolkit for bioinformatics analysis of custom high-throughput amplicon sequencing data\". <i>Molecular Ecology Resources<\/i> <b>17<\/b> (6): e234-e240. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2F1755-0998.12692\" target=\"_blank\">10.1111\/1755-0998.12692<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28544559\" target=\"_blank\">28544559<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PipeCraft%3A+Flexible+open-source+toolkit+for+bioinformatics+analysis+of+custom+high-throughput+amplicon+sequencing+data&rft.jtitle=Molecular+Ecology+Resources&rft.aulast=Anslan%2C+S.%3B+Bahram%2C+M.%3B+Hiirsalu%2C+I.+et+al.&rft.au=Anslan%2C+S.%3B+Bahram%2C+M.%3B+Hiirsalu%2C+I.+et+al.&rft.date=2017&rft.volume=17&rft.issue=6&rft.pages=e234-e240&rft_id=info:doi\/10.1111%2F1755-0998.12692&rft_id=info:pmid\/28544559&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Vetrovsk.C3.BDSeed18-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Vetrovsk.C3.BDSeed18_19-0\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-Vetrovsk.C3.BDSeed18_19-1\">19.1<\/a><\/sup> <sup><a href=\"#cite_ref-Vetrovsk.C3.BDSeed18_19-2\">19.2<\/a><\/sup> <sup><a href=\"#cite_ref-Vetrovsk.C3.BDSeed18_19-3\">19.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vetrovsk\u00fd, T.; Baldrian, P.; Morais, D. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6022770\" target=\"_blank\">\"SEED 2: A user-friendly platform for amplicon high-throughput sequencing data analyses\"<\/a>. <i>Bioinformatics<\/i> <b>34<\/b> (13): 2292-2294. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbty071\" target=\"_blank\">10.1093\/bioinformatics\/bty071<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6022770\/\" target=\"_blank\">PMC6022770<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29452334\" target=\"_blank\">29452334<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6022770\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6022770<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SEED+2%3A+A+user-friendly+platform+for+amplicon+high-throughput+sequencing+data+analyses&rft.jtitle=Bioinformatics&rft.aulast=Vetrovsk%C3%BD%2C+T.%3B+Baldrian%2C+P.%3B+Morais%2C+D.&rft.au=Vetrovsk%C3%BD%2C+T.%3B+Baldrian%2C+P.%3B+Morais%2C+D.&rft.date=2018&rft.volume=34&rft.issue=13&rft.pages=2292-2294&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbty071&rft_id=info:pmc\/PMC6022770&rft_id=info:pmid\/29452334&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6022770&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AmstutzCommon16-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AmstutzCommon16_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-AmstutzCommon16_20-1\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Amstutz, P.; Andeer, R.; Chapman, B. et al. (15 March 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/figshare.com\/articles\/Common_Workflow_Language_draft_3\/3115156\/1\" target=\"_blank\">\"Common Workflow Language, Draft 3\"<\/a>. <i>FigShare<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/figshare.com\/articles\/Common_Workflow_Language_draft_3\/3115156\/1\" target=\"_blank\">https:\/\/figshare.com\/articles\/Common_Workflow_Language_draft_3\/3115156\/1<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Common+Workflow+Language%2C+Draft+3&rft.atitle=FigShare&rft.aulast=Amstutz%2C+P.%3B+Andeer%2C+R.%3B+Chapman%2C+B.+et+al.&rft.au=Amstutz%2C+P.%3B+Andeer%2C+R.%3B+Chapman%2C+B.+et+al.&rft.date=15+March+2016&rft_id=https%3A%2F%2Ffigshare.com%2Farticles%2FCommon_Workflow_Language_draft_3%2F3115156%2F1&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HillionUsing17-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HillionUsing17_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hillion KH1, Kuzmin I2, Khodak A. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5747335\" target=\"_blank\">\"Using bio.tools to generate and annotate workbench tool descriptions\"<\/a>. <i>F1000Research<\/i> <b>6<\/b>: ELIXIR-2074. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.12688%2Ff1000research.12974.1\" target=\"_blank\">10.12688\/f1000research.12974.1<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5747335\/\" target=\"_blank\">PMC5747335<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29333231\" target=\"_blank\">29333231<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5747335\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5747335<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+bio.tools+to+generate+and+annotate+workbench+tool+descriptions&rft.jtitle=F1000Research&rft.aulast=Hillion+KH1%2C+Kuzmin+I2%2C+Khodak+A.+et+al.&rft.au=Hillion+KH1%2C+Kuzmin+I2%2C+Khodak+A.+et+al.&rft.date=2017&rft.volume=6&rft.pages=ELIXIR-2074&rft_id=info:doi\/10.12688%2Ff1000research.12974.1&rft_id=info:pmc\/PMC5747335&rft_id=info:pmid\/29333231&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5747335&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AmmarMS-EmpiRe19-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AmmarMS-EmpiRe19_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-AmmarMS-EmpiRe19_22-1\">22.1<\/a><\/sup> <sup><a href=\"#cite_ref-AmmarMS-EmpiRe19_22-2\">22.2<\/a><\/sup> <sup><a href=\"#cite_ref-AmmarMS-EmpiRe19_22-3\">22.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ammar, C.; Gruber, M.; Csaba, G. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6731086\" target=\"_blank\">\"MS-EmpiRe Utilizes Peptide-level Noise Distributions for Ultra-sensitive Detection of Differentially Expressed Proteins\"<\/a>. <i>Molecular and Cellular Proteomics<\/i> <b>18<\/b> (9): 1880\u201392. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1074%2Fmcp.RA119.001509\" target=\"_blank\">10.1074\/mcp.RA119.001509<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6731086\/\" target=\"_blank\">PMC6731086<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31235637\" target=\"_blank\">31235637<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6731086\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6731086<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MS-EmpiRe+Utilizes+Peptide-level+Noise+Distributions+for+Ultra-sensitive+Detection+of+Differentially+Expressed+Proteins&rft.jtitle=Molecular+and+Cellular+Proteomics&rft.aulast=Ammar%2C+C.%3B+Gruber%2C+M.%3B+Csaba%2C+G.+et+al.&rft.au=Ammar%2C+C.%3B+Gruber%2C+M.%3B+Csaba%2C+G.+et+al.&rft.date=2019&rft.volume=18&rft.issue=9&rft.pages=1880%E2%80%9392&rft_id=info:doi\/10.1074%2Fmcp.RA119.001509&rft_id=info:pmc\/PMC6731086&rft_id=info:pmid\/31235637&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6731086&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RuanFast19-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RuanFast19_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-RuanFast19_23-1\">23.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ruan, J.; Li, H. (2019). \"Fast and accurate long-read assembly with wtdbg2\". <i>Nature Methods<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41592-019-0669-3\" target=\"_blank\">10.1038\/s41592-019-0669-3<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31819265\" target=\"_blank\">31819265<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fast+and+accurate+long-read+assembly+with+wtdbg2&rft.jtitle=Nature+Methods&rft.aulast=Ruan%2C+J.%3B+Li%2C+H.&rft.au=Ruan%2C+J.%3B+Li%2C+H.&rft.date=2019&rft_id=info:doi\/10.1038%2Fs41592-019-0669-3&rft_id=info:pmid\/31819265&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BankevichSPAdes12-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BankevichSPAdes12_24-0\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-BankevichSPAdes12_24-1\">24.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bankevich, A.; Nurk, S.; Antipov, D. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3342519\" target=\"_blank\">\"SPAdes: A new genome assembly algorithm and its applications to single-cell sequencing\"<\/a>. <i>Journal of Computational Biology<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fcmb.2012.0021\" target=\"_blank\">10.1089\/cmb.2012.0021<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3342519\/\" target=\"_blank\">PMC3342519<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22506599\" target=\"_blank\">22506599<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3342519\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3342519<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SPAdes%3A+A+new+genome+assembly+algorithm+and+its+applications+to+single-cell+sequencing&rft.jtitle=Journal+of+Computational+Biology&rft.aulast=Bankevich%2C+A.%3B+Nurk%2C+S.%3B+Antipov%2C+D.+et+al.&rft.au=Bankevich%2C+A.%3B+Nurk%2C+S.%3B+Antipov%2C+D.+et+al.&rft.date=2012&rft_id=info:doi\/10.1089%2Fcmb.2012.0021&rft_id=info:pmc\/PMC3342519&rft_id=info:pmid\/22506599&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3342519&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuntCirculator15-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HuntCirculator15_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-HuntCirculator15_25-1\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hunt, M.; Silva, N.D.; Otto, T.D. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4699355\" target=\"_blank\">\"Circlator: Automated circularization of genome assemblies using long sequencing reads\"<\/a>. <i>Genome Biology<\/i> <b>16<\/b>: 294. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13059-015-0849-0\" target=\"_blank\">10.1186\/s13059-015-0849-0<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4699355\/\" target=\"_blank\">PMC4699355<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26714481\" target=\"_blank\">26714481<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4699355\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4699355<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Circlator%3A+Automated+circularization+of+genome+assemblies+using+long+sequencing+reads&rft.jtitle=Genome+Biology&rft.aulast=Hunt%2C+M.%3B+Silva%2C+N.D.%3B+Otto%2C+T.D.+et+al.&rft.au=Hunt%2C+M.%3B+Silva%2C+N.D.%3B+Otto%2C+T.D.+et+al.&rft.date=2015&rft.volume=16&rft.pages=294&rft_id=info:doi\/10.1186%2Fs13059-015-0849-0&rft_id=info:pmc\/PMC4699355&rft_id=info:pmid\/26714481&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4699355&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LangmeadUltra09-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LangmeadUltra09_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Langmead, B.; Trapnell, C.; Pop, M. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2690996\" target=\"_blank\">\"Ultrafast and memory-efficient alignment of short DNA sequences to the human genome\"<\/a>. <i>Genome Biology<\/i> <b>10<\/b> (3): R25. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2009-10-3-r25\" target=\"_blank\">10.1186\/gb-2009-10-3-r25<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2690996\/\" target=\"_blank\">PMC2690996<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19261174\" target=\"_blank\">19261174<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2690996\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2690996<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ultrafast+and+memory-efficient+alignment+of+short+DNA+sequences+to+the+human+genome&rft.jtitle=Genome+Biology&rft.aulast=Langmead%2C+B.%3B+Trapnell%2C+C.%3B+Pop%2C+M.+et+al.&rft.au=Langmead%2C+B.%3B+Trapnell%2C+C.%3B+Pop%2C+M.+et+al.&rft.date=2009&rft.volume=10&rft.issue=3&rft.pages=R25&rft_id=info:doi\/10.1186%2Fgb-2009-10-3-r25&rft_id=info:pmc\/PMC2690996&rft_id=info:pmid\/19261174&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2690996&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LangmeadFast12-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LangmeadFast12_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Langmead, B.; Salzberg, S.L. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3322381\" target=\"_blank\">\"Fast gapped-read alignment with Bowtie 2\"<\/a>. <i>Mature Methods<\/i> <b>9<\/b> (4): 357\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnmeth.1923\" target=\"_blank\">10.1038\/nmeth.1923<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3322381\/\" target=\"_blank\">PMC3322381<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22388286\" target=\"_blank\">22388286<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3322381\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3322381<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fast+gapped-read+alignment+with+Bowtie+2&rft.jtitle=Mature+Methods&rft.aulast=Langmead%2C+B.%3B+Salzberg%2C+S.L.&rft.au=Langmead%2C+B.%3B+Salzberg%2C+S.L.&rft.date=2012&rft.volume=9&rft.issue=4&rft.pages=357%E2%80%939&rft_id=info:doi\/10.1038%2Fnmeth.1923&rft_id=info:pmc\/PMC3322381&rft_id=info:pmid\/22388286&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3322381&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiFast09-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiFast09_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H.; Durbin, R. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234\" target=\"_blank\">\"Fast and accurate short read alignment with Burrows-Wheeler transform\"<\/a>. <i>Bioinformatics<\/i> <b>25<\/b> (14): 1754\u201360. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp324\" target=\"_blank\">10.1093\/bioinformatics\/btp324<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2705234\/\" target=\"_blank\">PMC2705234<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19451168\" target=\"_blank\">19451168<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fast+and+accurate+short+read+alignment+with+Burrows-Wheeler+transform&rft.jtitle=Bioinformatics&rft.aulast=Li%2C+H.%3B+Durbin%2C+R.&rft.au=Li%2C+H.%3B+Durbin%2C+R.&rft.date=2009&rft.volume=25&rft.issue=14&rft.pages=1754%E2%80%9360&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp324&rft_id=info:pmc\/PMC2705234&rft_id=info:pmid\/19451168&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2705234&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KorenCanu17-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KorenCanu17_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Koren, S.; Walenz, B.P.; Berlin, K. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5411767\" target=\"_blank\">\"Canu: Scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation\"<\/a>. <i>Genome Research<\/i> <b>27<\/b> (5): 722\u201336. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.215087.116\" target=\"_blank\">10.1101\/gr.215087.116<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5411767\/\" target=\"_blank\">PMC5411767<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28298431\" target=\"_blank\">28298431<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5411767\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5411767<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Canu%3A+Scalable+and+accurate+long-read+assembly+via+adaptive+k-mer+weighting+and+repeat+separation&rft.jtitle=Genome+Research&rft.aulast=Koren%2C+S.%3B+Walenz%2C+B.P.%3B+Berlin%2C+K.+et+al.&rft.au=Koren%2C+S.%3B+Walenz%2C+B.P.%3B+Berlin%2C+K.+et+al.&rft.date=2017&rft.volume=27&rft.issue=5&rft.pages=722%E2%80%9336&rft_id=info:doi\/10.1101%2Fgr.215087.116&rft_id=info:pmc\/PMC5411767&rft_id=info:pmid\/28298431&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5411767&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiaoFeature14-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiaoFeature14_30-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiaoFeature14_30-1\">30.1<\/a><\/sup> <sup><a href=\"#cite_ref-LiaoFeature14_30-2\">30.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liao, Y.; Smyth, G.K.; Shi, W. (2014). \"featureCounts: an efficient general purpose program for assigning sequence reads to genomic features\". <i>Bioinformatics<\/i> <b>30<\/b> (7): 923\u201330. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtt656\" target=\"_blank\">10.1093\/bioinformatics\/btt656<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24227677\" target=\"_blank\">24227677<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=featureCounts%3A+an+efficient+general+purpose+program+for+assigning+sequence+reads+to+genomic+features&rft.jtitle=Bioinformatics&rft.aulast=Liao%2C+Y.%3B+Smyth%2C+G.K.%3B+Shi%2C+W.&rft.au=Liao%2C+Y.%3B+Smyth%2C+G.K.%3B+Shi%2C+W.&rft.date=2014&rft.volume=30&rft.issue=7&rft.pages=923%E2%80%9330&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtt656&rft_id=info:pmid\/24227677&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Sovi.C4.87Fast16-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Sovi.C4.87Fast16_31-0\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-Sovi.C4.87Fast16_31-1\">31.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sovi\u0107, I.; \u0160iki\u0107, M.; Wilm, A. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4835549\" target=\"_blank\">\"Fast and sensitive mapping of nanopore sequencing reads with GraphMap\"<\/a>. <i>Nature Communications<\/i> <b>7<\/b>: 11307. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fncomms11307\" target=\"_blank\">10.1038\/ncomms11307<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4835549\/\" target=\"_blank\">PMC4835549<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27079541\" target=\"_blank\">27079541<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4835549\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4835549<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fast+and+sensitive+mapping+of+nanopore+sequencing+reads+with+GraphMap&rft.jtitle=Nature+Communications&rft.aulast=Sovi%C4%87%2C+I.%3B+%C5%A0iki%C4%87%2C+M.%3B+Wilm%2C+A.+et+al.&rft.au=Sovi%C4%87%2C+I.%3B+%C5%A0iki%C4%87%2C+M.%3B+Wilm%2C+A.+et+al.&rft.date=2016&rft.volume=7&rft.pages=11307&rft_id=info:doi\/10.1038%2Fncomms11307&rft_id=info:pmc\/PMC4835549&rft_id=info:pmid\/27079541&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4835549&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimGraph19-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KimGraph19_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, D.; Paggi, J.M.; Park, C. et al. (2019). \"Graph-based genome alignment and genotyping with HISAT2 and HISAT-genotype\". <i>Nature Biotechnology<\/i> <b>37<\/b> (8): 907\u201315. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41587-019-0201-4\" target=\"_blank\">10.1038\/s41587-019-0201-4<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31375807\" target=\"_blank\">31375807<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Graph-based+genome+alignment+and+genotyping+with+HISAT2+and+HISAT-genotype&rft.jtitle=Nature+Biotechnology&rft.aulast=Kim%2C+D.%3B+Paggi%2C+J.M.%3B+Park%2C+C.+et+al.&rft.au=Kim%2C+D.%3B+Paggi%2C+J.M.%3B+Park%2C+C.+et+al.&rft.date=2019&rft.volume=37&rft.issue=8&rft.pages=907%E2%80%9315&rft_id=info:doi\/10.1038%2Fs41587-019-0201-4&rft_id=info:pmid\/31375807&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WheelerNHMMER13-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WheelerNHMMER13_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wheeler, T.J.; Eddy, S.R. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3777106\" target=\"_blank\">\"nhmmer: DNA homology search with profile HMMs\"<\/a>. <i>Bioinformatics<\/i> <b>29<\/b> (19): 2487-9. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtt403\" target=\"_blank\">10.1093\/bioinformatics\/btt403<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3777106\/\" target=\"_blank\">PMC3777106<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23842809\" target=\"_blank\">23842809<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3777106\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3777106<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=nhmmer%3A+DNA+homology+search+with+profile+HMMs&rft.jtitle=Bioinformatics&rft.aulast=Wheeler%2C+T.J.%3B+Eddy%2C+S.R.&rft.au=Wheeler%2C+T.J.%3B+Eddy%2C+S.R.&rft.date=2013&rft.volume=29&rft.issue=19&rft.pages=2487-9&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtt403&rft_id=info:pmc\/PMC3777106&rft_id=info:pmid\/23842809&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3777106&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangPure19-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangPure19_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, Q.; Ni, C.-m., Li, Z. et al. (2019). \"PureseqTM: efficient and accurate prediction of transmembrane topology from amino acid sequence only\". <i>bioRxiv<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2F627307\" target=\"_blank\">10.1101\/627307<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PureseqTM%3A+efficient+and+accurate+prediction+of+transmembrane+topology+from+amino+acid+sequence+only&rft.jtitle=bioRxiv&rft.aulast=Wang%2C+Q.%3B+Ni%2C+C.-m.%2C+Li%2C+Z.+et+al.&rft.au=Wang%2C+Q.%3B+Ni%2C+C.-m.%2C+Li%2C+Z.+et+al.&rft.date=2019&rft_id=info:doi\/10.1101%2F627307&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShenRMATS14-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShenRMATS14_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shen, S.; Park, J.W.; Lu, Z.X. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4280593\" target=\"_blank\">\"rMATS: Robust and flexible detection of differential alternative splicing from replicate RNA-Seq data\"<\/a>. <i>Proceedings of the National Academy of Sciences of the United States of America<\/i> <b>111<\/b> (51). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1073%2Fpnas.1419161111\" target=\"_blank\">10.1073\/pnas.1419161111<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4280593\/\" target=\"_blank\">PMC4280593<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25480548\" target=\"_blank\">25480548<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4280593\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4280593<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=rMATS%3A+Robust+and+flexible+detection+of+differential+alternative+splicing+from+replicate+RNA-Seq+data&rft.jtitle=Proceedings+of+the+National+Academy+of+Sciences+of+the+United+States+of+America&rft.aulast=Shen%2C+S.%3B+Park%2C+J.W.%3B+Lu%2C+Z.X.+et+al.&rft.au=Shen%2C+S.%3B+Park%2C+J.W.%3B+Lu%2C+Z.X.+et+al.&rft.date=2014&rft.volume=111&rft.issue=51&rft_id=info:doi\/10.1073%2Fpnas.1419161111&rft_id=info:pmc\/PMC4280593&rft_id=info:pmid\/25480548&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4280593&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RehmsmeierFast04-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RehmsmeierFast04_36-0\">36.0<\/a><\/sup> <sup><a href=\"#cite_ref-RehmsmeierFast04_36-1\">36.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rehmsmeier, M.; Steffen, P.; Hochsmann, M. et al. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4280593\" target=\"_blank\">\"Fast and effective prediction of microRNA\/target duplexes\"<\/a>. <i>RNA<\/i> <b>10<\/b> (10). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1261%2Frna.5248604\" target=\"_blank\">10.1261\/rna.5248604<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4280593\/\" target=\"_blank\">PMC4280593<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15383676\" target=\"_blank\">15383676<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4280593\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4280593<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fast+and+effective+prediction+of+microRNA%2Ftarget+duplexes&rft.jtitle=RNA&rft.aulast=Rehmsmeier%2C+M.%3B+Steffen%2C+P.%3B+Hochsmann%2C+M.+et+al.&rft.au=Rehmsmeier%2C+M.%3B+Steffen%2C+P.%3B+Hochsmann%2C+M.+et+al.&rft.date=2004&rft.volume=10&rft.issue=10&rft_id=info:doi\/10.1261%2Frna.5248604&rft_id=info:pmc\/PMC4280593&rft_id=info:pmid\/15383676&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4280593&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Li-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Li_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, B.; Dewey, C.N. (323). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163565\" target=\"_blank\">\"RSEM: Accurate transcript quantification from RNA-Seq data with or without a reference genome\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-323\" target=\"_blank\">10.1186\/1471-2105-12-323<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3163565\/\" target=\"_blank\">PMC3163565<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21816040\" target=\"_blank\">21816040<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163565\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163565<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=RSEM%3A+Accurate+transcript+quantification+from+RNA-Seq+data+with+or+without+a+reference+genome&rft.jtitle=BMC+Bioinformatics&rft.aulast=Li%2C+B.%3B+Dewey%2C+C.N.&rft.au=Li%2C+B.%3B+Dewey%2C+C.N.&rft.date=323&rft.volume=12&rft_id=info:doi\/10.1186%2F1471-2105-12-323&rft_id=info:pmc\/PMC3163565&rft_id=info:pmid\/21816040&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3163565&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiTheSeq09-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiTheSeq09_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H.; Handsaker, B.; Wysoker, A. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002\" target=\"_blank\">\"The Sequence Alignment\/Map format and SAMtools\"<\/a>. <i>Bioinformatics<\/i> <b>25<\/b> (16): 2078-9. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp352\" target=\"_blank\">10.1093\/bioinformatics\/btp352<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2723002\/\" target=\"_blank\">PMC2723002<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19505943\" target=\"_blank\">19505943<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Sequence+Alignment%2FMap+format+and+SAMtools&rft.jtitle=Bioinformatics&rft.aulast=Li%2C+H.%3B+Handsaker%2C+B.%3B+Wysoker%2C+A.+et+al.&rft.au=Li%2C+H.%3B+Handsaker%2C+B.%3B+Wysoker%2C+A.+et+al.&rft.date=2009&rft.volume=25&rft.issue=16&rft.pages=2078-9&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp352&rft_id=info:pmc\/PMC2723002&rft_id=info:pmid\/19505943&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2723002&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiMini18-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiMini18_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6137996\" target=\"_blank\">\"Minimap2: pairwise alignment for nucleotide sequences\"<\/a>. <i>Bioinformatics<\/i> <b>34<\/b> (18): 3094\u2013100. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbty191\" target=\"_blank\">10.1093\/bioinformatics\/bty191<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6137996\/\" target=\"_blank\">PMC6137996<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29750242\" target=\"_blank\">29750242<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6137996\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6137996<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Minimap2%3A+pairwise+alignment+for+nucleotide+sequences&rft.jtitle=Bioinformatics&rft.aulast=Li%2C+H.&rft.au=Li%2C+H.&rft.date=2018&rft.volume=34&rft.issue=18&rft.pages=3094%E2%80%93100&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbty191&rft_id=info:pmc\/PMC6137996&rft_id=info:pmid\/29750242&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6137996&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiMini16-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiMini16_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4937194\" target=\"_blank\">\"Minimap and miniasm: Fast mapping and de novo assembly for noisy long sequences\"<\/a>. <i>Bioinformatics<\/i> <b>32<\/b> (14): 2103\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtw152\" target=\"_blank\">10.1093\/bioinformatics\/btw152<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4937194\/\" target=\"_blank\">PMC4937194<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27153593\" target=\"_blank\">27153593<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4937194\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4937194<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Minimap+and+miniasm%3A+Fast+mapping+and+de+novo+assembly+for+noisy+long+sequences&rft.jtitle=Bioinformatics&rft.aulast=Li%2C+H.&rft.au=Li%2C+H.&rft.date=2016&rft.volume=32&rft.issue=14&rft.pages=2103%E2%80%9310&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtw152&rft_id=info:pmc\/PMC4937194&rft_id=info:pmid\/27153593&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4937194&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VaserFast17-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VaserFast17_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vaser, R.; Sovi\u0107, I.; Nagarajan, N. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5411768\" target=\"_blank\">\"Fast and accurate de novo genome assembly from long uncorrected reads\"<\/a>. <i>Genome Research<\/i> <b>27<\/b> (5): 737-46. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.214270.116\" target=\"_blank\">10.1101\/gr.214270.116<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5411768\/\" target=\"_blank\">PMC5411768<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28100585\" target=\"_blank\">28100585<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5411768\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5411768<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fast+and+accurate+de+novo+genome+assembly+from+long+uncorrected+reads&rft.jtitle=Genome+Research&rft.aulast=Vaser%2C+R.%3B+Sovi%C4%87%2C+I.%3B+Nagarajan%2C+N.+et+al.&rft.au=Vaser%2C+R.%3B+Sovi%C4%87%2C+I.%3B+Nagarajan%2C+N.+et+al.&rft.date=2017&rft.volume=27&rft.issue=5&rft.pages=737-46&rft_id=info:doi\/10.1101%2Fgr.214270.116&rft_id=info:pmc\/PMC5411768&rft_id=info:pmid\/28100585&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5411768&rfr_id=info:sid\/en.wikipedia.org:Journal:From_command-line_bioinformatics_to_bioGUI\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204348\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.145 seconds\nReal time usage: 3.364 seconds\nPreprocessor visited node count: 35475\/1000000\nPreprocessor generated node count: 38354\/1000000\nPost\u2010expand include size: 325927\/2097152 bytes\nTemplate argument size: 99696\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 760.558 1 - -total\n 87.77% 667.577 1 - Template:Reflist\n 75.10% 571.173 39 - Template:Cite_journal\n 74.90% 569.640 41 - Template:Citation\/core\n 13.06% 99.309 110 - Template:Citation\/identifier\n 5.32% 40.430 1 - Template:Infobox_journal_article\n 5.08% 38.625 1 - Template:Infobox\n 4.95% 37.647 42 - Template:Citation\/make_link\n 4.90% 37.238 253 - Template:Hide_in_print\n 3.47% 26.364 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11355-0!*!0!!en!5!* and timestamp 20200707204345 and revision id 37327\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:From_command-line_bioinformatics_to_bioGUI\">https:\/\/www.limswiki.org\/index.php\/Journal:From_command-line_bioinformatics_to_bioGUI<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","41e81da03f2f60115a39008f3e196345_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/1f\/Fig1_Joppich_PeerJ2019_7.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/77\/Fig2_Joppich_PeerJ2019_7.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/18\/FigA1_Joppich_PeerJ2019_7.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3d\/FigA2_Joppich_PeerJ2019_7.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a9\/FigA3_Joppich_PeerJ2019_7.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/16\/FigA4_Joppich_PeerJ2019_7.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/7d\/FigA5_Joppich_PeerJ2019_7.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/ff\/FigA6_Joppich_PeerJ2019_7.jpg"],"41e81da03f2f60115a39008f3e196345_timestamp":1594154625,"2351ba0d1aa807936bc9922c2403b7f2_type":"article","2351ba0d1aa807936bc9922c2403b7f2_title":"ROBOT: A Tool for Automating Ontology Workflows (Jackson et al. 2019)","2351ba0d1aa807936bc9922c2403b7f2_url":"https:\/\/www.limswiki.org\/index.php\/Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows","2351ba0d1aa807936bc9922c2403b7f2_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:ROBOT: A Tool for Automating Ontology Workflows\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nROBOT: A Tool for Automating Ontology WorkflowsJournal\n \nBMC BioinformaticsAuthor(s)\n \nJackson, Rebecca C.; Balhoff, James, P.; Douglass, Eric; Harris, Nomi L.; Mungall, Christopher J.; Overton, James A.Author affiliation(s)\n \nKnocean, Inc.; University of North Carolina; Lawrence Berkeley National LaboratoryPrimary contact\n \nEmail: via SpringerLinkYear published\n \n2019Volume and issue\n \n20Page(s)\n \n407DOI\n \n10.1186\/s12859-019-3002-3ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/link.springer.com\/article\/10.1186\/s12859-019-3002-3Download\n \nhttps:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-019-3002-3.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Implementation \n\n3.1 Overview \n3.2 Architecture \n3.3 Commands and operations \n\n3.3.1 Conversion \n3.3.2 Reasoning \n\n\n3.4 Working with external ontologies \n\n3.4.1 Extractions \n3.4.2 Removal and filtration \n3.4.3 Merging \n\n\n3.5 Querying and reporting \n\n3.5.1 Querying \n3.5.2 Verification \n3.5.3 Reporting \n3.5.4 Repairs \n\n\n3.6 Templated ontology development \n3.7 Workflows \n3.8 The Ontology Development Kit \n\n\n4 Results and discussion \n\n4.1 Ontology for Biomedical Investigations \n4.2 Disease Ontology \n\n\n5 Conclusions \n6 Software availability and requirements \n7 Abbreviations \n8 Acknowledgements \n\n8.1 Contributions \n8.2 Funding \n8.3 Availability of data and materials \n8.4 Competing interests \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nBackground: Ontologies are invaluable in the life sciences, but building and maintaining ontologies often requires a challenging number of distinct tasks such as running automated reasoners and quality control checks, extracting dependencies and application-specific subsets, generating standard reports, and generating release files in multiple formats. Similar to more general software development, automation is the key to executing and managing these tasks effectively and to releasing more robust products in standard forms.\nFor ontologies using the Web Ontology Language (OWL), the OWL API (application programming interface) Java library is the foundation for a range of software tools, including the Prot\u00e9g\u00e9 ontology editor. In the Open Biological and Biomedical Ontologies (OBO) community, we recognized the need to package a wide range of low-level OWL API functionality into a library of common higher-level operations and to make those operations available as a command-line tool.\nResults: ROBOT (a recursive acronym for \u201cROBOT is an OBO Tool\u201d) is an open-source library and command-line tool for automating ontology development tasks. The library can be called from any programming language that runs on the Java Virtual Machine (JVM). Most usage is through the command-line tool, which runs on macOS, Linux, and Windows. ROBOT provides ontology processing commands for a variety of tasks, including commands for converting formats, running a reasoner, creating import modules, running reports, and various other tasks. These commands can be combined into larger workflows using a separate task execution system such as GNU Make, and workflows can be automatically executed within continuous integration systems.\nConclusions: ROBOT supports automation of a wide range of ontology development tasks, focusing on OBO conventions. It packages common high-level ontology development functionality into a convenient library and makes it easy to configure, combine, and execute individual tasks in comprehensive, automated workflows. This helps ontology developers to efficiently create, maintain, and release high-quality ontologies so they can spend more time focusing on development tasks. It also helps guarantee released ontologies are free of certain types of logical errors and conform to standard quality control checks, increasing the overall robustness and efficiency of the ontology development lifecycle.\nKeywords: ontology development, automation, ontology release, reasoning, workflows, quality control, import management\n\nBackground \nOntologies are vital parts of the informatics ecosystem, supporting life science research, enabling analysis of high-throughput datasets, data standardization and integration, search, and discovery. However, there is a lack of tools supporting the complete ontology development lifecycle, especially when compared with the software development lifecycle. This has resulted in many groups developing their own ad hoc ontology development workflows, often with time-consuming and inefficient manual steps. In some cases, groups release ontologies without any kind of systematic workflow or quality control process, which can result in errors or problems with downstream applications or analyses.\nNoy et al. (2010) describe a general ontology lifecycle, with a focus on bio-ontologies.[1] First, requirements for the ontology are gathered. Then, the ontology is collaboratively developed in an ontology editor such as Prot\u00e9g\u00e9.[2] Once the requirements have been fulfilled, the ontology is published and feedback is solicited from the community. Feedback is integrated back into development, and the ontology is continuously updated and released. At any point after the initial publication, the ontology may be deployed in other applications.\nIn broad strokes, this ontology development lifecycle reflects much of our experience of ontology development in the Open Biological and Biomedical Ontologies (OBO) community[3], circa 2010. A wide range of Semantic Web-based software exists to support these steps, including many tools for Web Ontology Language (OWL) ontology development. In practice, though, the OBO community has relied predominantly on the free and open-source Prot\u00e9g\u00e9 OWL editor for manual editing and conversion, and on a small set of other tools supporting OBO conventions.\nOther than Prot\u00e9g\u00e9, the most prominent suite of tools used by the OBO community has been the Onto-animal suite, developed by the He Group[4], which includes Ontobee[5], Ontofox[6], and Ontorat.[7] These tools are free web services backed by a Virtuoso triplestore loaded with the latest version of all available OBO community ontologies, as well as some other ontologies. Ontobee is an ontology term browser. Ontofox implements the MIREOT term extraction method.[8] Ontorat implements template-based ontology term creation. Together with a few other tools, these support an extensible ontology development strategy[9] covering a range of ontology development tasks, many of which can combined and automated using a sequence of web-based application programming interface (API) calls.\nThe core operations of the Onto-Animal suite are driven by SPARQL queries against the centralized triplestore. This results in a number of limitations. First, only the specific version of each ontology loaded into that triplestore can be used. This is a particularly severe limitation during ontology development. Second, processing is done on the centralized server, limiting the processing power available to any user. Third, SPARQL has limited utility when working with OWL logical axioms.\nThese limitations are mitigated by running software locally, loading the desired versions of the desired ontologies, and using OWL API[10] for OWL-native processing. A number of tools used in the OBO community have done precisely this. We have seen a spectrum of development, from tools that are focused on a single project, to tools used by a dozen related projects, to the current push for tools that are shared across the OBO community.\nSlimmer[11], created as part of the eNanoMapper ontology project[12], uses OWL API to create ontology subsets (also known as \u201cslims\u201d). A configuration file allows the user to specify which terms to include and which annotations to include on those terms. OntoPilot[13], developed for the Plant Phenotype Ontology, uses OWL API via Jython (a version of Python that runs on the Java Virtual Machine) to provide an integrated ontology development framework, including term imports, term creation, releases, and documentation.\nThe lack of automation seen circa 2010 led directly to a lack of standardization, with each ontology editor or group adopting a slightly different approach to manual editing in Prot\u00e9g\u00e9. This diversity of practices, even within the OBO community, made it a challenge to develop tools to serve multiple ontology projects. OWLTools[14] was designed for use by multiple OBO ontology projects, providing convenience methods on top of the OWL API. OWLTools includes the OBO Ontology Release Tool (OORT)[15], a command-line tool to release OWL- and OBO-format ontologies. OORT provides a series of basic commands to create a release pipeline for an ontology, including module extraction with MIREOT, support for multiple input ontologies, reasoning, and creation of \"main\" and \"simple\" release products.\nROBOT (a recursive acronym for \u201cROBOT is an OBO Tool\u201d) was developed to replace OWLTools and OORT with a more modular and maintainable code base. It builds on previous experience to include a comprehensive set of automation capabilities to support an even wider range of OBO projects. Development began in 2015 and continues today, with more than 1000 commits from a dozen contributors. ROBOT is freely available open-source software. Although we do not track our users, a recent GitHub search shows that at least 26 ontology projects in the OBO community have adopted ROBOT.\n\nImplementation \nOverview \nROBOT provides a standardized yet configurable way to support the ontology development lifecycle via a library of common high-level functionality and a command-line interface. ROBOT builds on OWL API and is compatible with all ontology syntaxes that OWL API supports: RDF\/XML, OWL\/XML, Turtle, OWL Functional Syntax, OWL Manchester Syntax, and OBO format. The source code is written in Java and is available from our GitHub repository[16] under an open source (BSD 3) license. It is also released as a Java library on Maven Central. ROBOT code can be used from any programming language that runs on the Java Virtual Machine (JVM). The command-line tool is packaged as a JAR file that can be run on Unix (including macOS and Linux), Windows, and other platforms supported by the JVM. This JAR file is available for download from the ROBOT GitHub site[16], along with platform-specific scripts for using ROBOT from the command line. Installation instructions and documentation are available from the OBO Library.\n\nArchitecture \nWe previously described the basic architecture of the tool[17], which we summarize here.\nThe ROBOT source code consists of two parts: \u2018robot-core\u2019 and \u2018robot-command.\u2019 \u2018robot-core\u2019 is a library supporting common ontology development tasks, which we call \u201coperations.\u201d \u2018robot-command\u2019 provides a command-line interface divided into \u201ccommands,\u201d each of which wraps a \u2018robot-core\u2019 operation.\nMost ROBOT operations package low-level functionality provided by OWL API into high-level functionality common to ontology development workflows in the OBO community. For best compatibility, we aim to match the exact version of OWL API used by ROBOT with the exact version used by the latest Prot\u00e9g\u00e9 release. Some operations use Apache Jena.[18] Each operation works with Java objects that represent OWL ontologies, OWL reasoners, OWL classes, etc., while each command works with command-line option strings and files. The commands also perform various conversion and validation steps. The command-line interface uses the Apache Commons CLI library[19] for parsing commands.\nEach operation has a set of unit tests, built with JUnit[20], that are executed each time the final product (the JAR file) is generated. Each command in ROBOT is documented in its own web page (e.g., http:\/\/robot.obolibrary.org\/reason). The web pages are authored in Markdown format and contain embedded command-line examples that are parsed and executed as part of our integration tests, with the results compared against a \u201cgold standard\u201d set of outputs. ROBOT\u2019s \u2018diff\u2019 functionality is used when comparing ontology files, otherwise standard file comparison is used. This helps ensure correctness and consistency of documentation and code. The unit tests and integration tests are executed on any pull request onto the codebase via Travis continuous integration (Travis CI)[21] so that contributions to the codebase are verified.\n\nCommands and operations \nROBOT currently provides 15 operations (in the \u2018robot-core\u2019 library) and 19 commands (for the command-line interface). Some commands are quite specialized, and most ontology projects will not make use of all of them. Here we provide an overview of the most common and general commands. In each case, the core functionality is supported by operations in the \u2018robot-core\u2019 library, that can be used independently of the command-line interface from any programming language that runs on the JVM.\n\nConversion \nA variety of OWL ontology formats are supported, including RDF\/XML, Turtle, Manchester, OBO format, and more. To enable further interoperability, ROBOT includes a convert command to change between supported ontology formats. A complete list of supported formats can be found in the convert documentation.\n\nReasoning \nReasoning is one of the most important operations in ROBOT. The reason command covers two uses: logical validation of an ontology and automatic classification. In both cases, users can choose their preferred reasoner, which is used to perform inference. Large ontologies such as the Gene Ontology typically use ELK[22], which performs reasoning quickly using the OWL EL profile. Smaller ontologies with richer axiomatization, such as the Relations Ontology, typically use a complete OWL DL reasoner such as HermiT.[23]\nWhen the reason command is invoked on an input ontology, ROBOT will initiate a reasoner using the OWL API Reasoner interface. The resulting inferences are checked to ensure the ontology is logically coherent: the ontology must be consistent and have no unsatisfiable classes (i.e., classes that cannot be instantiated without introducing an inconsistency). If the ontology is incoherent, this is reported and execution halts. ROBOT can optionally perform additional checks, such as ensuring that no two classes are inferred to be equivalent post-reasoning.\nIf the ontology is consistent, ROBOT will perform automatic classification. All direct inferred \u2018subClassOf\u2019 axioms are added to the ontology. Generation of other types of axioms can be configured.\nThe assertion of all inferred axioms is often a fundamental step in the release process for biomedical ontologies. Many of these ontology classes only assert a single named superclass (\u2018A subClassOf B\u2019, where B is another class in the ontology), and zero or more anonymous superclasses and\/or anonymous equivalent classes (\u2018A subClassOf\/equivalentTo (R some B)\u2019, where R is an object property). These anonymous classes allow the reasoner to make inferences, which are then asserted. Therefore, in the release version of an ontology, a class may have more than one named superclass.\nThe reason command has additional \u201chelper\u201d commands. The relax command asserts entailed subClassOf axioms according to a simple structural rule: an expression \u2018A equivalentTo (R some B) and \u2026\u2019 entails \u2018A subClassOf R some B\u2019. This can be useful as consumers of bio-ontologies often expect to navigate these expressions, e.g., partonomy in GO and Uberon. The relax command relieves the ontology developer from the need to assert these in addition to the equivalence axioms, and as such it is also often included in release workflows. Finally, the reduce command removes redundant \u2018subClassOf\u2019 axioms, and can be used after relax to remove duplicate axioms that were asserted in that step.\nThe materialize command uses the Expression Materializing Reasoner (EMR) extension to assert inferred expressions of the form \u2018A subClassOf R some B\u2019.[24] Where the reason command asserts inferred named superclasses, materialize asserts anonymous superclasses. This is not part of the standard release cycle but can be beneficial for creating complete ontology subsets.\n\nWorking with external ontologies \nThe OBO Foundry aims to coordinate ontologies in a modular fashion, such that parts of some ontologies can be used as building blocks for other ontologies. For example, the ChEBI chemical entities ontology[25] is used to construct OWL definitions for metabolic processes and activities in the Gene Ontology.[26] There are a variety of different strategies for leveraging external ontologies and managing dependencies between ontologies, depending on the use case.\n\nExtractions \nThe extract command creates a module based on a set of entities to extract (the \u201cseed\u201d). There are four different extraction methods (as specified by the --method option): MIREOT, TOP, BOT, and STAR.\nROBOT\u2019s MIREOT extraction method is based on the principle of the same name[8] and requires that one or more \u201cbottom\u201d entities are specified. Optionally, one or more \u201ctop\u201d entities can also be specified. The command extracts all the \u201cbottom\u201d level entities and their ancestors up to the \u201ctop\u201d level from the input ontology. If no \u201ctop\u201d entities are provided, ancestors up to the top-level entity (\u2018owl: Thing\u2019) are included.\nThe TOP, BOT, and STAR methods make use of the OWL API Syntactic Locality Module Extraction (SLME) implementation, which is guaranteed to capture all information logically relevant to the seed set.[27] The BOT method (\u201cbottom\u201d) includes all relationships between the input entities and their ancestors. The TOP method includes all relationships between the input entities and their descendants. Finally, the STAR method only includes all relationships between input entities. The STAR method produces the smallest outputs, while the TOP method typically produces the largest outputs.\nIn order to support ontology term provenance, the extract command has an --annotate-with-source true option that will annotate each extracted term with the URL of the source ontology that it is extracted from.\n\nRemoval and filtration \nThe remove and filter commands are used for fine-grained operations on OWL axioms. remove allows users to choose which sets of axioms they wish to remove from a target ontology. filter does the opposite, so that only selected axioms are copied from the input into a new output ontology.\nThese two commands work by starting with a seed set of entities, then applying various selectors to find related entities, and finally selecting which axiom types to remove or filter. We expect only a small number of \u201cpower users\u201d to use this feature directly, but these commands will eventually provide a foundation for other higher-level commands.\nThese commands can be used to generate ontology subsets based on annotations by either filtering for or removing entities with the specified annotation. OBO Foundry ontologies often annotate classes with the \u2018in subset\u2019 property to specify where a class might be used. The annotation selector allows a user to provide a full annotation value or a pattern to match using regular expression.\n\nMerging \nThe merge command combines two or more separate input ontologies into a single ontology. It also provides the ability to merge all imported ontologies of a single input ontology into one main ontology, which is often used when creating a release.\nMerging imported ontologies (specified by import statements) into the input ontology is performed automatically so that the user does not need to list each imported ontology as an input. We offer the option (--collapse-import-closure false) to turn this feature off, supporting cases in which users may merge multiple input ontologies that have import statements but want to keep their imports separate.\n\nQuerying and reporting \nOntology workflows typically include query operations over the ontology, producing reports which may be informative for both editors and users of the ontology, e.g., a table of all classes plus their textual definitions. Query operations can also be used for validation checks. The SPARQL query language provides a universal and declarative way for ontology maintainers to create ontology reports and validation checks.[28] ROBOT provides a convenient way to perform queries with the query command, or validation checks using verify. Additionally, the report command includes a configurable package of standard queries for OBO projects that can be used in any ontology workflow, without requiring the maintainer to be familiar with SPARQL.\n\nQuerying \nROBOT\u2019s query command runs SPARQL queries on ontologies or other RDF resources. This can be used by an ontology maintainer to either perform interactive queries, or more typically to include standard queries into an ontology workflow. The query command wraps one of the few operations that uses Apache Jena[18], rather than OWL API. The Jena API allows ROBOT to load an ontology as a collection of triples contained by an RDF Model object. It provides a SPARQL query engine for those models, which we use to run all queries.\n\u2018SPARQL SELECT\u2019 queries produce a comma- or tab-separated table of results. ASK queries produce a file with a Boolean value. \u2018SPARQL CONSTRUCT\u2019 queries produce an RDF file, which can be further processed by ROBOT or merged back into the loaded ontology. \u2018CONSTRUCT\u2019s provide a convenient way of performing \u201cmacro\u201d style expansion.[29] \u2018SPARQL UPDATE\u2019 queries insert and\/or remove data directly in an ontology (as an RDF Model). ROBOT converts the updated RDF Model back to an OWL API ontology object to be saved in any of the supported syntaxes.\nThe query command supports an option to load imported ontologies as named graphs with the --use-graphs option. If this is set to true, the imports can be queried as named graphs (the name being that ontology\u2019s IRI) and the default graph is a union of all graphs. Using the default graph is similar to conducting a merge of all the imports prior to querying, but the distinction between imports would be lost in a merge.\n\nVerification \nThe verify command is a variation on the \u2018SPARQL SELECT\u2019 execution. The queries are used to ensure that an ontology conforms to a predetermined set of conditions, by, for example, ensuring that no class has multiple textual definitions. Given a SELECT query, verify will succeed (i.e., exit with status code 0) if no results are returned. It will fail (i.e., exit with a non-zero status code) if any results are return from the query. So, given a SPARQL query that SELECTs for invalid data, the verify command will verify that the ontology (or other resource) does not contain any such invalid data.\n\nReporting \nThe report command is an extension of query and verify that provides a series of configurable quality control (QC) checks for an ontology and returns a spreadsheet or YAML output of the violations. The spreadsheet is output in either comma- or tab-separated format and easy for a user to read, while the YAML output can be easily parsed by other programs.\nThe QC checks include annotation checks, logical checks, and metadata checks. Annotations are important to facilitate human comprehension, so the report command finds cases where missing or duplicate annotations could cause problems. Logical checks look at the structural coherency and consistency of the ontology. Finally, report identifies missing ontology metadata, as specified by OBO Foundry recommendations.\nThere are three levels of violations that are reported: ERROR, WARN, and INFO. An ERROR is the most severe, such as a missing or duplicate label. By default, the report command fails if there are any ERROR-level violations, halting any automated build processes. These types of violations must be fixed before publishing an ontology. WARN-level violations should be fixed as soon as possible, e.g., inferred one-to-one class equivalencies, which are typically unintended in OBO projects. INFO is for recommended fixes that help maintain consistency across OBO Foundry ontologies, such as beginning a definition with an uppercase letter and ending with a period. report can be configured with a command-line option to fail on a different violation level or to never fail, regardless of any violations. We document each QC check with a suggestion for a manual fix that the user can apply.\nA default \u201cprofile\u201d with report levels for each QC check is provided by ROBOT, but users are also able to create their own profiles. In these profiles, they can change the violation levels of individual checks, choose to exclude certain checks, and add their own checks as SPARQL queries. For example, some ontologies may categorize a class lacking a textual definition as an error, while others may categorize this as a warning. One of our goals is to converge on a standard profile that is maximally useful for the set of all ontologies in the OBO library, encouraging adoption of common quality control checks.\n\nRepairs \nAlthough most problems raised by validate and report must be fixed manually, ROBOT also provides a repair command that can automatically fix certain problems. The current implementation will merge annotations on duplicate axioms and update references to deprecated classes when they are annotated with a suggested replacement. We intend to extend repair to a wider range of common problems for which the correct fix is clear.\n\nTemplated ontology development \nROBOT provides a template-driven ontology term generation system. Users also have the option to plug other term generation systems such as Dead Simple OWL Design Patterns (DOS-DPs)[30] into their workflow.\nA huge amount of data is stored in spreadsheets and databases, and tabular formats are well suited to many sorts of data. ROBOT\u2019s template command allows users to convert tabular data into RDF\/OWL format. A ROBOT template is simply a tab-separated values (TSV) or comma-separated values (CSV) file with some special conventions, which are outlined in the ROBOT template documentation.\nThese templates can be used for modular ontology development. The template spreadsheets may be maintained as part of the ontology\u2019s source code repository, and instead of directly editing the ontology file, developers edit rows in the spreadsheet which correspond to terms in the ontology. The template command is then used to generate a module of the ontology, which is included as an import statement in the editors\u2019 version of the ontology and merged during the release process.\n\nWorkflows \nA workflow consists of a set of tasks coordinated by some workflow system. Ontology workflows consist of tasks such as executing QC checks, building import modules, reasoning over ontologies, and generating various ontology release products.\nROBOT itself is not a workflow manager, although it allows multiple commands to be chained together into one long command. When chaining ROBOT commands, the output ontology from one command is passed directly as the input to the next command. For example, chaining may be used to replace two commands that merge ontologies and then reason over the merged product:\n\n robot merge --input ont-1.owl --input ont-2.owl --output merged.owl\n robot reason --input merged.owl --output reasoned.owl\nInstead of creating the merged product and running reason over that, it can be done in one command:\n\n robot merge --input ont-1.owl --input ont-2.owl reason --output reasoned.owl\nThe key advantage to chaining is that ontologies do not have to be serialized and parsed between each step; the same OWL API ontology object is maintained in memory and passed through the chain of ROBOT commands. For large ontologies, chaining can vastly improve ROBOT\u2019s performance.\nBecause ROBOT commands can be executed on the command line, a number of different workflow systems can be used. We highlight the use of GNU Make[31], which is typically used to compile software. A Makefile consists of a set of rules used to make \u201ctargets.\u201d In ontology development, the Makefile is used for automated tasks such as preparing an ontology for release. In this case, the targets are usually ontology files. The \u201crecipes\u201d for the rules are Unix-style system commands, carried out by the make command.\nROBOT commands can be used as the \u201crecipes\u201d to make the \u201ctargets.\u201d A typical workflow will not use all 19 of the ROBOT commands. For example, not all ontology projects may use ROBOT templates and therefore not all release workflows need to include the template command. Ontology developers can decide which commands are needed to perform the release and build a workflow around those commands. Figure 1 shows a standard way in which a selection of ROBOT commands is combined for a release workflow.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. The ROBOT release workflow. A typical release workflow using ROBOT. The ontology edit file ONT-edit.owl is first verified as a quality control check with ROBOT verify. Then, text files containing lists of external ontology terms in the imports directory are used to regenerate import modules using extract, ensuring that the imports are up-to-date. ONT-edit.owl is then passed through a series of ROBOT commands (reason, relax, reduce, and annotate) to generate the release, ONT.owl. Finally, ONT.owl is converted to OBO format.\n\n\n\nFirst, quality control checks are run over the editors\u2019 version of the ontology with report or verify. These look for equivalent classes, trailing whitespace in annotations, self-references, incorrect cross-reference syntax, and missing labels. The results are saved to a specified \"reports\/\" directory. If there are any ERROR-level violations, the task will fail and write the violations to a table so that they can be easily identified. This step allows developers to quickly see if new changes have introduced any problems within the ontology and fix them before releasing.\nAssuming the initial QC check step has completed successfully, the next step is to create the import modules. The ROBOT extract is run for each entry in a list of imports, which have corresponding term files (for the seed set) in the \"imports\/\" directory. This creates all the import modules in the same \"imports\/\" directory. This ensures that when an ontology is released with external terms, all external terms are up-to-date with the released versions of the source ontologies. Releasing out-of-date external terms can cause confusion, as the term will show both the old and new details in ontology search services like Ontobee[5] and the Ontology Lookup Service.[32] Additional QC checks can be run over the full ontology with imports using the verify command or by running report again.\nLast, the main release products are created: the OWL file and the OBO file. To create the OWL release, the editors\u2019 file is passed through a series of chained ROBOT commands: reason, relax, reduce, and annotate. This series of commands helps to ensure that the released ontology is both easy to browse and understand, as well as free of any redundant axioms. If any of these commands fail, the Make process will terminate with the corresponding error message. For example, if an ontology is incoherent the reason step will fail. Finally, the <\/tt>annotate<\/tt> command adds the version IRI to the ontology metadata. This OWL file is then converted to OBO format, at which point all targets are copied to a dated release directory.\n\nThe Ontology Development Kit \nCreating a Makefile to coordinate all these steps can be challenging. We make this easier for ontology developers by providing an Ontology Development Kit (ODK).[33] This can be used to create a GitHub repository following a standard layout, with a standard Makefile following the workflow detailed above. The resulting GitHub repository will also be automatically configured to run the validation steps (such as report) of the workflow via Travis CI.[21] The workflow can also be executed using Docker with ODK containers released on DockerHub.[34] This allows easy execution of workflows on either the local computer of an ontology developer, with Travis CI, or through scalable-build tools such as Jenkins.[33]\nODK builds on ROBOT and demonstrates ROBOT\u2019s utility, but a full discussion is beyond the scope of this article.\n\nResults and discussion \nWhile there are many other Semantic Web and OWL development tools available, a number of factors has driven the OBO community to build and support ROBOT. First, the OBO commitment to open-source development is a strong reason to use open-source software. Second, our wide reliance on the free and open Prot\u00e9g\u00e9 editor is a strong reason to use the same OWL API library that it is built upon. Third, there is a strong incentive to pool our limited resources and invest in shared tooling. Fourth, the OBO community has a range of conventions that support interoperability, and our workflows are simpler if we build these assumptions into the tools. Points three and four are clearly in tension: what is the right balance to strike between reusing general Semantic Web tools and building our own? Prot\u00e9g\u00e9, OWL API, and various OWL reasoners are general tools that we use, for instance, but we have strong conventions in our community for identifiers, release artifacts, metadata, quality control, etc. that these tools do not help us to enforce. Our compromise is to reuse open-source tools as much as practical and invest community resources in customizing general tools such as OWL API to serve the needs of our community. A growing number of ontology developers are using ROBOT to help automate their quality-checking and release workflows. Two case studies are described here.\n\nOntology for Biomedical Investigations \nThe Ontology for Biomedical Investigations (OBI) is an OBO Foundry ontology that aims to describe the processes, agents, devices, inputs, and outputs of scientific investigations.[35] When the project began more than a decade ago, development was done in Prot\u00e9g\u00e9, without any automation, and hosted on SourceForge. Today, OBI uses ROBOT to implement an automated workflow, supported by GitHub pull requests and Travis CI testing. More than 50 people have contributed to OBI development, including two of the authors of this paper (Overton and Jackson).\nOBI has always imported a range of terms from other OBO projects, and OBI developers have maintained a number of separate OWL files to facilitate concurrent development by different groups of developers. When it comes time to prepare a new release of OBI, the various OWL files must be merged, tested, and reasoned over.\nIn the early days of the project, OBI developer Alan Ruttenberg wrote a series of scripts for quality control and common operations, but merging, reasoning, and testing a new release still involved many hours of work by OBI developers. In 2013, James Overton developed a precursor to ROBOT: an automated build tool written in Java, using OWL API and Apache Ant, that automated some of the build, test, and release processes. This drastically reduced the time required to make a release, allowing for more frequent releases. While this code was specific to OBI workflows, some of it was used in early versions of ROBOT.\nIn 2017, OBI moved from SourceForge to GitHub, and the release workflow was updated to use ROBOT throughout. This change vastly increased the degree of automation for ontology development tasks, expanded capabilities, and allowed OBI to pool some of its development resources with the wider OBO community to support shared tooling. OBI currently uses a Makefile that defines a range of tasks for managing imports, converting templates, merging, reasoning, testing, and releasing new versions of OBI. The Makefile specifies various target files, and most target files are generated from a single ROBOT command or a single chain of ROBOT commands. The key steps are:\n\n Update imports from upstream ontologies (currently using Ontofox[6]). OBI imports subsets of terms from more than a dozen OBO projects. As discussed, ROBOT supports this functionality with extract, but OBI\u2019s use of Ontofox predates ROBOT development and has not yet been migrated.\n Normalize RDF\/XML for cleaner history of changes in the version control system (\u2018robot convert\u2019). Different versions of OWL API have slightly different serialization behavior, which can lead to spurious reports of differences that make it more difficult to see relevant changes to the source files.\n Convert template files (TSV) to OWL modules (\u2018robot template\u2019). Templates often make it easier for domain experts to contribute to ontology development and enforce ontology design patterns that improve the overall quality of OBI.\n Merge imports and templates with \u2018obi-edit.owl\u2019 (\u2018robot merge\u2019). OBI uses a number of import and template files to enforce a separation of concerns, rather than making all changes in a single source file. These are merged into a single release file.\n Use \u2018SPARQL CONSTRUCT\u2019 queries to update various term annotations (\u2018robot query\u2019). Some standard term metadata can be automatically added and updated, rather than manually maintained.\n Run an automated test suite (\u2018robot verify\u2019). A range of quality control checks helps to ensure that errors have not been introduced into OBI by recent changes.\n Run the HermiT reasoner (\u2018robot reason\u2019). Reasoning checks the logical consistency of OBI and performs automated classification of terms.\n Update annotations for release (\u2018robot annotate\u2019). These annotations include the dated version IRI of this release of OBI.\n Extract the OBI Core subset (\u2018robot extract\u2019). The OBI Core subset provides approximately 100 important terms for educational and documentation purposes.\n Create a list of OBI terms (\u2018robot query\u2019). The term list is used to report on the new terms added to OBI with each release.\nDisease Ontology \nThe Disease Ontology (DO) is an OBO Foundry ontology that provides a standardized description of human diseases, including the phenotypic characteristics, symptoms, genetic bases, and related medical terminology. These terms are used by various model organism databases to provide a consistent representation of diseases.[36] The DO is developed at the University of Maryland School of Medicine by Lynn Schriml and her team, which currently includes one of the authors (Jackson).\nIn order to accurately and thoroughly describe the different aspects of diseases, DO makes use of more than 10 other biological ontologies. In the past, all DO imports were manually created and maintained. This led to inconsistencies as ontologies were updated and expanded, and also made it very difficult to add new entities to the imports.\nIn 2018, DO switched their entire automated build process (contained in the Makefile) from OWLTools[14] and OORT[15] to ROBOT. Instead of manually updating import modules, all required entities are now specified in text files. When a developer wishes to add a new imported entity, they simply add a line to the text file and run make imports. All imports are automatically regenerated during releases, as well, to keep the information up-to-date.\nBefore ROBOT, the monthly DO releases took multiple hours to run and required additional hours of manual editing and review. Now, DO developers simply run the make release command and all content is generated in less than 20\u2009minutes. The release process makes use of ROBOT commands such as report to run quality control checks over the release products and reduce the time spent reviewing output.\nBoth ROBOT verify and <\/tt>report<\/tt> are used for DO\u2019s Travis CI system.[21] Each time a new commit is pushed to the GitHub repository, a series of queries is run against the ontology files to ensure they meet certain standards. If they do not (or if report fails with an ERROR-level violation), Travis CI notifies developers that the checks have failed with a red \u201cX\u201d next to the commit. Clicking on the red \u201cX\u201d will take the user to Travis CI, where they can see the command log to determine what caused the failure. On success, a green check mark is displayed next to the commit to show that the checks have passed.\n\nConclusions \nROBOT makes it easy for ontology developers to automate a wide range of tedious and error-prone development tasks, freeing their time to focus on other important parts of the ontology life cycle. Circa 2010, most OBO projects were manually edited, with manual imports, manually tested, and manually released using only Prot\u00e9g\u00e9. With ROBOT, ontology developers have a convenient tool for building powerful workflows that include format conversion, reasoning, extracting, querying, updating, testing, reporting, templating, and more. Low-level OWL API and Apache Jena operations are packaged into a library of high-level operations, and these operations are wrapped in a convenient command-line interface that is supported on the common server and desktop platforms. With ODK, developers benefit not only from ROBOT, but additionally from a wide range of best practices and standard procedures developed and shared by the OBO community.\nROBOT is open-source software developed by a community of a dozen contributors with more than 1000 commits, hundreds of closed issues, and six releases. The ROBOT source code is freely available on GitHub and Maven Central. Documentation for the library is available on Javadoc.io, and documentation for the commands is available on our website, where you will also find examples of usage, test files, and explanations of common errors.\nWith ROBOT, we have tried to strike a balance between general tools such as Prot\u00e9g\u00e9 and the specific needs of the OBO community, and to deliver the benefits of automation from software development to ontology development.\n\nSoftware availability and requirements \nProject name: ROBOT (ROBOT is an OBO Tool)\nProject home page: http:\/\/robot.obolibrary.org\/\nOperating system(s): Platform-independent\nProgramming language: Java 8\nOther requirements: None for the command-line tool. The ROBOT library depends on the following: Apache Jena, SnakeYAML, OpenCSV, FasterXML Jackson, OWL API, Apache Commons IO, Apache Maven, JSONLD-JAVA, Prot\u00e9g\u00e9, JUnit, SLF4J, and fmt-maven-plugin.\nLicense: ROBOT is available under BSD 3. Dependencies are available under Apache 2.0 (Jena, SnakeYAML, OpenCSV, Jackson, OWL API, Commons IO, and Maven), BSD 3 (JSONLD-JAVA and Prot\u00e9g\u00e9), EPL-1.0 (JUnit), and the MIT License (SLF4J and fmt-maven-plugin).\nAny restrictions to use by non-academics: None other than those specified by the licenses.\n\nAbbreviations \nCSV: comma-separated values\nDL: description logic\nDO: Disease Ontology\nDOS-DP: Dead Simple OWL Design Patterns\nEMR: Expression Materializing Reasoner\nGUI: graphical user interface\nOBI: Ontology for Biomedical Investigations\nOBO: Open Biological Ontologies\nODK: Ontology Development Kit\nOWL: Web Ontology Language\nROBOT: ROBOT is an OBO Tool\nTSV: tab-separated values\n\nAcknowledgements \nWe thank David Osumi-Sutherland for requirements and testing and Nico Matentzoglu for submitting bug reports and feature requests. We also thank all of our users and everyone who has participated in ROBOT development, from contributing code and documentation, to reporting bugs and requesting features.\n\nContributions \nRCJ, JAO and JPB wrote the manuscript. NH edited the manuscript. RCJ, JPB, ED, JAO and CJM developed and implemented the software. All authors read and approved the final manuscript.\n\nFunding \nThis work and related resources are supported by the following grants:\n\n From the Department of Health and Human Services (NIH), for \u201cImmune Epitope Database and Analysis Resource Program\u201d (HHSN272201200010C)\n From the National Institute of Allergy and Infectious Diseases (NIH), for \u201cHuman immune signatures of Dengue virus and Mycobacterium Tuberculosis exposure in infection, disease and vaccination\u201d (1-U19-AI-118626-01)\n From the National Human Genome Research Institute (NIH), for \u201cServices to support the OBO foundry standards\u201d (R24-HG010032)\n From the National Human Genome Research Institute (NIH), for \u201cGene Ontology Consortium\u201d (U41HG002273)\nThese funding bodies did not play any role in the design of ROBOT, the interpretation of data, or the writing of this manuscript.\n\nAvailability of data and materials \nAll code is available from https:\/\/github.com\/ontodev\/robot\/.\n\nCompeting interests \nThe authors declare that they have no competing interests. ROBOT development is largely funded by the grants acknowledged above. Knocean, Inc. is subcontracted for ROBOT development from those grants. Overton is president and owner of Knocean, Inc. Jackson is a contractor for Knocean, Inc.\n\nReferences \n\n\n\u2191 Noy, N.; Tudorache, T.; Nyulas, C. et al. (2010). \"The ontology life cycle: Integrated tools for editing, publishing, peer review, and evolution of ontologies\". AMIA Annual Symposium Proceedings 2010: 552\u20136. PMC PMC3041389. PMID 21347039. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3041389 .   \n\n\u2191 Horridge, M.; Tsarkov, D. (2006). \"Supporting early adoption of OWL 1.1 with Prot\u00e9g\u00e9-OWL and FaCT++\". OWL: Experiences and Directions 2006: 1\u20137. http:\/\/webont.org\/owled\/2006\/accepted06.html .   \n\n\u2191 Smith, B.; Ashburner, M.; Rosse, C. et al. (2007). \"The OBO Foundry: Coordinated evolution of ontologies to support biomedical data integration\". Nature Biotechnology 25 (11): 1251\u20135. doi:10.1038\/nbt1346. PMC PMC2814061. PMID 17989687. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2814061 .   \n\n\u2191 He, Y.; Zheng, J.; Lin, Y. (2015). \"Onto-animal tools for reusing ontologies, generating and editing ontology terms, and dereferencing ontology terms\". Proceedings of the 2015 International Conference on Biomedical Ontology: 1\u20132. http:\/\/ceur-ws.org\/Vol-1515\/ .   \n\n\u2191 5.0 5.1 Xiang, Z.; Mungall, C.; Ruttenberg, A. et al. (2012). \"Ontobee: A Linked Data Server and Browser for Ontology Terms\". Proceedings of the Second International Conference on Biomedical Ontology: 279\u201381. http:\/\/ceur-ws.org\/Vol-833\/ .   \n\n\u2191 6.0 6.1 Xiang, Z.; Courtot, M.; Brinkman, R.R. et al. (2010). \"OntoFox: Web-based support for ontology reuse\". BMC Research Notes 3: 175. doi:10.1186\/1756-0500-3-175. PMC PMC2911465. PMID 20569493. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2911465 .   \n\n\u2191 Xiang, Z.; Zheng, J.; Lin. Y. et al. (2015). \"Ontorat: Automatic generation of new ontology terms, annotations, and axioms based on ontology design patterns\". Journal of Biomedical Semantics 6: 4. doi:10.1186\/2041-1480-6-4. PMC PMC4362828. PMID 25785185. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4362828 .   \n\n\u2191 8.0 8.1 Courtot, M.; Gibson, F.; Lister, A. L. et al. (2011). \"MIREOT: The minimum information to reference an external ontology term\". Applied Ontology 6 (1): 23\u201333. doi:10.3233\/AO-2011-0087.   \n\n\u2191 He, Y.; Xiang, Z.; Zheng, J. et al. (2018). \"The eXtensible ontology development (XOD) principles and tool implementation to support ontology interoperabilitye\". Journal of Biomedical Semantics 9: 3. doi:10.1186\/s13326-017-0169-2.   \n\n\u2191 Horridge, M.; Bechhofer, S.; Noppens, O. (2007). \"Igniting the OWL 1.1 Touch Paper: The OWL API\". OWL: Experiences and Directions 2007: 1\u20139. http:\/\/webont.org\/owled\/2007\/Proceedings.html .   \n\n\u2191 \"enanomapper \/ slimmer\". GitHub. https:\/\/github.com\/enanomapper\/slimmer\/ . Retrieved 21 May 2019 .   \n\n\u2191 Hastings, J.; Jeliazkova, N.; Owen, G. et al. (2015). \"eNanoMapper: harnessing ontologies to enable data integration for nanomaterial risk assessment\". Journal of Biomedical Semantics 6: 10. doi:10.1186\/s13326-015-0005-5. PMC PMC4374589. PMID 25815161. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4374589 .   \n\n\u2191 Stucky, B.J.; Luc, A. (2017). \"OntoPilot: New Software to Simplify and Accelerate Ontology Development and Deployment\". Biodiversity Information Science and Standards 1: e20192. doi:10.3897\/tdwgproceedings.1.20192.   \n\n\u2191 14.0 14.1 \"owlcollab \/ owltools\". GitHub. https:\/\/github.com\/owlcollab\/owltools . Retrieved 21 May 2019 .   \n\n\u2191 15.0 15.1 \"owlcollab \/ owltools : Oort Intro\". GitHub. https:\/\/github.com\/owlcollab\/owltools\/wiki\/Oort-Intro . Retrieved 21 May 2019 .   \n\n\u2191 16.0 16.1 \"ontodev \/ robot\". GitHub. https:\/\/github.com\/ontodev\/robot . Retrieved 09 October 2018 .   \n\n\u2191 Overton, J.A.; Dietze, H.; Essaid, S. et al. (2015). \"ROBOT: A command-line tool for ontology development\". Proceedings of the 2015 International Conference on Biomedical Ontology: 1\u20132. http:\/\/ceur-ws.org\/Vol-1515\/ .   \n\n\u2191 18.0 18.1 Carroll, J.J.; Dickinson. I.; Dollin, C. et al. (2004). \"Jena: Implementing the semantic web recommendations\". WWW Alt. '04: Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters: 74\u201383. doi:10.1145\/1013367.1013381.   \n\n\u2191 \"Commons CLI\". Apache Commons. https:\/\/commons.apache.org\/proper\/commons-cli\/ . Retrieved 23 May 2019 .   \n\n\u2191 \"JUnit\". junit.org. https:\/\/junit.org\/junit4\/ . Retrieved 21 May 2019 .   \n\n\u2191 21.0 21.1 21.2 \"Travis CI\". Travis CI, GmbH. https:\/\/travis-ci.org\/ . Retrieved 21 May 2019 .   \n\n\u2191 Kazakov, Y.; Kr\u00f6tzsch, M.; Siman\u010d\u00edk, F. (2014). \"The Incredible ELK\". Journal of Automated Reasoning 53: 1\u201361. doi:10.1007\/s10817-013-9296-3.   \n\n\u2191 Shearer, R.; Motik, B.; Horrocks, I. (2008). \"HermiT: A Highly-Efficient OWL Reasoner\". OWL: Experiences and Directions 2008: 1\u201310. http:\/\/ceur-ws.org\/Vol-432\/ .   \n\n\u2191 \"owlcollab \/ expression-materializing-reasoner\". GitHub. https:\/\/github.com\/owlcollab\/expression-materializing-reasoner . Retrieved 09 October 2018 .   \n\n\u2191 Hastings, J.; de Matos, P.; Dekker, A. et al. (2013). \"The ChEBI reference database and ontology for biologically relevant chemistry: Enhancements for 2013\". Nucleic Acids Research 41 (DB1): D456-63. doi:10.1093\/nar\/gks1146. PMC PMC3531142. PMID 23180789. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531142 .   \n\n\u2191 Hill, D.P.; Adams, N.; Bada, M.A. et al. (2013). \"Dovetailing biology and chemistry: Integrating the Gene Ontology with the ChEBI chemical ontology\". BMC Genomics 14: 513. doi:10.1186\/1471-2164-14-513. PMC PMC3733925. PMID 23895341. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3733925 .   \n\n\u2191 Grau, B.C.; Horrocks, I.; Kazakov, Y. et al. (2008). \"Modular Reuse of Ontologies: Theory and Practice\". Journal of Artificial Intelligence Research 31: 273\u2013318. doi:10.1613\/jair.2375.   \n\n\u2191 Harris, S.; Seaborne, A.; Prud'hommeaux, E., ed. (21 March 2013). \"SPARQL 1.1 Query Language\". w3.org. https:\/\/www.w3.org\/TR\/sparql11-query\/ . Retrieved 23 May 2019 .   \n\n\u2191 Mungall, C.; Ruttenberg, A.; Osumi-Sutherland, D. (2011). \"Taking shortcuts with OWL using safe macros\". Nature Precedings. doi:10.1038\/npre.2011.5292.2.   \n\n\u2191 Osumi-Sutherland, D.; Courtot, M.; Balhoff, J.P. et al. (2017). \"Dead simple OWL design pattern\". Journal of Biomedical Semantics 8 (1): 18. doi:10.1186\/s13326-017-0126-0. PMC PMC5460348. PMID 28583177. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5460348 .   \n\n\u2191 \"GNU Make\". GNU - Software. https:\/\/www.gnu.org\/software\/make\/ . Retrieved 21 May 2019 .   \n\n\u2191 C\u00f4t\u00e9, R.; Reisinger, F.; Martens, L. et al. (2010). \"The Ontology Lookup Service: Bigger and better\". Nucleic Acids Research 38 (WSI): W155-60. doi:10.1093\/nar\/gkq331. PMC PMC2896109. PMID 20460452. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2896109 .   \n\n\u2191 33.0 33.1 \"INCATools \/ ontology-development-kit\". GitHub. https:\/\/github.com\/INCATools\/ontology-development-kit\/ . Retrieved 09 October 2018 .   \n\n\u2191 \"obolibrary\". DockerHub. https:\/\/hub.docker.com\/u\/obolibrary . Retrieved 09 October 2018 .   \n\n\u2191 Bandrowski, A.; Brinkman, R.; Brochhausen, M. et al. (2016). \"The Ontology for Biomedical Investigations\". PLoS One 11 (4): e0154556. doi:10.1371\/journal.pone.0154556. PMC PMC4851331. PMID 27128319. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4851331 .   \n\n\u2191 Bello, S.M.; Shimoyama, M.; Mitraka, E. et al. (2018). \"Disease Ontology: improving and unifying disease annotations across species\". Disease Models and Mechanisms 11 (3): dmm032839. doi:10.1242\/dmm.032839. PMC PMC5897730. PMID 29590633. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5897730 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference. A few citations were turned into simple inline links to content, where appropriate.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\">https:\/\/www.limswiki.org\/index.php\/Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 17 February 2020, at 22:21.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 5,112 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","2351ba0d1aa807936bc9922c2403b7f2_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_ROBOT_A_Tool_for_Automating_Ontology_Workflows skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:ROBOT: A Tool for Automating Ontology Workflows<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: <a href=\"https:\/\/www.limswiki.org\/index.php\/Ontology_(information_science)\" title=\"Ontology (information science)\" class=\"wiki-link\" data-key=\"52d0664bde4b458e81fbc128b911a4a6\">Ontologies<\/a> are invaluable in the life sciences, but building and maintaining ontologies often requires a challenging number of distinct tasks such as running automated reasoners and <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> checks, extracting dependencies and application-specific subsets, generating standard reports, and generating release files in multiple formats. Similar to more general <a href=\"https:\/\/www.limswiki.org\/index.php\/Systems_development_life_cycle\" title=\"Systems development life cycle\" class=\"wiki-link\" data-key=\"b96939e19621960ee123770c13fa1a84\">software development<\/a>, automation is the key to executing and managing these tasks effectively and to releasing more robust products in standard forms.\n<\/p><p>For ontologies using the Web Ontology Language (OWL), the OWL API (<a href=\"https:\/\/www.limswiki.org\/index.php\/Application_programming_interface\" title=\"Application programming interface\" class=\"wiki-link\" data-key=\"36fc319869eba4613cb0854b421b0934\">application programming interface<\/a>) Java library is the foundation for a range of software tools, including the Prot\u00e9g\u00e9 ontology editor. In the Open Biological and Biomedical Ontologies (OBO) community, we recognized the need to package a wide range of low-level OWL API functionality into a library of common higher-level operations and to make those operations available as a command-line tool.\n<\/p><p><b>Results<\/b>: ROBOT (a recursive acronym for \u201cROBOT is an OBO Tool\u201d) is an open-source library and command-line tool for automating ontology development tasks. The library can be called from any programming language that runs on the Java Virtual Machine (JVM). Most usage is through the command-line tool, which runs on macOS, Linux, and Windows. ROBOT provides ontology processing commands for a variety of tasks, including commands for converting formats, running a reasoner, creating import modules, running reports, and various other tasks. These commands can be combined into larger <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a> using a separate task execution system such as GNU Make, and workflows can be automatically executed within continuous integration systems.\n<\/p><p><b>Conclusions<\/b>: ROBOT supports automation of a wide range of ontology development tasks, focusing on OBO conventions. It packages common high-level ontology development functionality into a convenient library and makes it easy to configure, combine, and execute individual tasks in comprehensive, automated workflows. This helps ontology developers to efficiently create, maintain, and release high-quality ontologies so they can spend more time focusing on development tasks. It also helps guarantee released ontologies are free of certain types of logical errors and conform to standard quality control checks, increasing the overall robustness and efficiency of the ontology development lifecycle.\n<\/p><p><b>Keywords<\/b>: ontology development, automation, ontology release, reasoning, workflows, quality control, import management\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Ontology_(information_science)\" title=\"Ontology (information science)\" class=\"wiki-link\" data-key=\"52d0664bde4b458e81fbc128b911a4a6\">Ontologies<\/a> are vital parts of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics_(academic_field)\" title=\"Informatics (academic field)\" class=\"wiki-link\" data-key=\"0391318826a5d9f9a1a1bcc88394739f\">informatics<\/a> ecosystem, supporting life science research, enabling analysis of high-throughput datasets, data standardization and integration, search, and discovery. However, there is a lack of tools supporting the complete ontology development lifecycle, especially when compared with the <a href=\"https:\/\/www.limswiki.org\/index.php\/Systems_development_life_cycle\" title=\"Systems development life cycle\" class=\"wiki-link\" data-key=\"b96939e19621960ee123770c13fa1a84\">software development lifecycle<\/a>. This has resulted in many groups developing their own <i>ad hoc<\/i> ontology development <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a>, often with time-consuming and inefficient manual steps. In some cases, groups release ontologies without any kind of systematic workflow or <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> process, which can result in errors or problems with downstream applications or <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analyses<\/a>.\n<\/p><p>Noy <i>et al.<\/i> (2010) describe a general ontology lifecycle, with a focus on bio-ontologies.<sup id=\"rdp-ebb-cite_ref-NoyTheOnt10_1-0\" class=\"reference\"><a href=\"#cite_note-NoyTheOnt10-1\">[1]<\/a><\/sup> First, requirements for the ontology are gathered. Then, the ontology is collaboratively developed in an ontology editor such as Prot\u00e9g\u00e9.<sup id=\"rdp-ebb-cite_ref-HorridgeSupport06_2-0\" class=\"reference\"><a href=\"#cite_note-HorridgeSupport06-2\">[2]<\/a><\/sup> Once the requirements have been fulfilled, the ontology is published and feedback is solicited from the community. Feedback is integrated back into development, and the ontology is continuously updated and released. At any point after the initial publication, the ontology may be deployed in other applications.\n<\/p><p>In broad strokes, this ontology development lifecycle reflects much of our experience of ontology development in the Open Biological and Biomedical Ontologies (OBO) community<sup id=\"rdp-ebb-cite_ref-SmithTheOBO07_3-0\" class=\"reference\"><a href=\"#cite_note-SmithTheOBO07-3\">[3]<\/a><\/sup>, circa 2010. A wide range of Semantic Web-based software exists to support these steps, including many tools for Web Ontology Language (OWL) ontology development. In practice, though, the OBO community has relied predominantly on the free and open-source Prot\u00e9g\u00e9 OWL editor for manual editing and conversion, and on a small set of other tools supporting OBO conventions.\n<\/p><p>Other than Prot\u00e9g\u00e9, the most prominent suite of tools used by the OBO community has been the Onto-animal suite, developed by the He Group<sup id=\"rdp-ebb-cite_ref-HeOnto-Animal15_4-0\" class=\"reference\"><a href=\"#cite_note-HeOnto-Animal15-4\">[4]<\/a><\/sup>, which includes Ontobee<sup id=\"rdp-ebb-cite_ref-XiangOntobee11_5-0\" class=\"reference\"><a href=\"#cite_note-XiangOntobee11-5\">[5]<\/a><\/sup>, Ontofox<sup id=\"rdp-ebb-cite_ref-XiangOntoFox10_6-0\" class=\"reference\"><a href=\"#cite_note-XiangOntoFox10-6\">[6]<\/a><\/sup>, and Ontorat.<sup id=\"rdp-ebb-cite_ref-XiangOntoRat15_7-0\" class=\"reference\"><a href=\"#cite_note-XiangOntoRat15-7\">[7]<\/a><\/sup> These tools are free web services backed by a Virtuoso <a href=\"https:\/\/en.wikipedia.org\/wiki\/Triplestore\" class=\"extiw wiki-link\" title=\"wikipedia:Triplestore\" data-key=\"1caa8db58dd63bb4c31a8a8997abfa21\">triplestore<\/a> loaded with the latest version of all available OBO community ontologies, as well as some other ontologies. Ontobee is an ontology term browser. Ontofox implements the MIREOT term extraction method.<sup id=\"rdp-ebb-cite_ref-CourtotMIREOT11_8-0\" class=\"reference\"><a href=\"#cite_note-CourtotMIREOT11-8\">[8]<\/a><\/sup> Ontorat implements template-based ontology term creation. Together with a few other tools, these support an extensible ontology development strategy<sup id=\"rdp-ebb-cite_ref-HeTheExt18_9-0\" class=\"reference\"><a href=\"#cite_note-HeTheExt18-9\">[9]<\/a><\/sup> covering a range of ontology development tasks, many of which can combined and automated using a sequence of web-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Application_programming_interface\" title=\"Application programming interface\" class=\"wiki-link\" data-key=\"36fc319869eba4613cb0854b421b0934\">application programming interface<\/a> (API) calls.\n<\/p><p>The core operations of the Onto-Animal suite are driven by SPARQL queries against the centralized triplestore. This results in a number of limitations. First, only the specific version of each ontology loaded into that triplestore can be used. This is a particularly severe limitation during ontology development. Second, processing is done on the centralized server, limiting the processing power available to any user. Third, SPARQL has limited utility when working with OWL logical axioms.\n<\/p><p>These limitations are mitigated by running software locally, loading the desired versions of the desired ontologies, and using OWL API<sup id=\"rdp-ebb-cite_ref-HorridgeIgnit07_10-0\" class=\"reference\"><a href=\"#cite_note-HorridgeIgnit07-10\">[10]<\/a><\/sup> for OWL-native processing. A number of tools used in the OBO community have done precisely this. We have seen a spectrum of development, from tools that are focused on a single project, to tools used by a dozen related projects, to the current push for tools that are shared across the OBO community.\n<\/p><p>Slimmer<sup id=\"rdp-ebb-cite_ref-GitHubSlimmer_11-0\" class=\"reference\"><a href=\"#cite_note-GitHubSlimmer-11\">[11]<\/a><\/sup>, created as part of the eNanoMapper ontology project<sup id=\"rdp-ebb-cite_ref-HastingsEnano15_12-0\" class=\"reference\"><a href=\"#cite_note-HastingsEnano15-12\">[12]<\/a><\/sup>, uses OWL API to create ontology subsets (also known as \u201cslims\u201d). A configuration file allows the user to specify which terms to include and which annotations to include on those terms. OntoPilot<sup id=\"rdp-ebb-cite_ref-StuckyOnto17_13-0\" class=\"reference\"><a href=\"#cite_note-StuckyOnto17-13\">[13]<\/a><\/sup>, developed for the Plant Phenotype Ontology, uses OWL API via Jython (a version of Python that runs on the Java Virtual Machine) to provide an integrated ontology development framework, including term imports, term creation, releases, and documentation.\n<\/p><p>The lack of automation seen circa 2010 led directly to a lack of standardization, with each ontology editor or group adopting a slightly different approach to manual editing in Prot\u00e9g\u00e9. This diversity of practices, even within the OBO community, made it a challenge to develop tools to serve multiple ontology projects. OWLTools<sup id=\"rdp-ebb-cite_ref-GitHubOwlTools_14-0\" class=\"reference\"><a href=\"#cite_note-GitHubOwlTools-14\">[14]<\/a><\/sup> was designed for use by multiple OBO ontology projects, providing convenience methods on top of the OWL API. OWLTools includes the OBO Ontology Release Tool (OORT)<sup id=\"rdp-ebb-cite_ref-GitHubOwlTools-Oort_15-0\" class=\"reference\"><a href=\"#cite_note-GitHubOwlTools-Oort-15\">[15]<\/a><\/sup>, a command-line tool to release OWL- and OBO-format ontologies. OORT provides a series of basic commands to create a release pipeline for an ontology, including module extraction with MIREOT, support for multiple input ontologies, reasoning, and creation of \"main\" and \"simple\" release products.\n<\/p><p>ROBOT (a recursive acronym for \u201cROBOT is an OBO Tool\u201d) was developed to replace OWLTools and OORT with a more modular and maintainable code base. It builds on previous experience to include a comprehensive set of automation capabilities to support an even wider range of OBO projects. Development began in 2015 and continues today, with more than 1000 commits from a dozen contributors. ROBOT is freely available open-source software. Although we do not track our users, a recent GitHub search shows that at least 26 ontology projects in the OBO community have adopted ROBOT.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Overview\">Overview<\/span><\/h3>\n<p>ROBOT provides a standardized yet configurable way to support the ontology development lifecycle via a library of common high-level functionality and a command-line interface. ROBOT builds on OWL API and is compatible with all ontology syntaxes that OWL API supports: RDF\/XML, OWL\/XML, Turtle, OWL Functional Syntax, OWL Manchester Syntax, and OBO format. The source code is written in Java and is available from our GitHub repository<sup id=\"rdp-ebb-cite_ref-GitHubROBOT_16-0\" class=\"reference\"><a href=\"#cite_note-GitHubROBOT-16\">[16]<\/a><\/sup> under an open source (BSD 3) license. It is also released as a Java library on Maven Central. ROBOT code can be used from any programming language that runs on the Java Virtual Machine (JVM). The command-line tool is packaged as a JAR file that can be run on Unix (including macOS and Linux), Windows, and other platforms supported by the JVM. This JAR file is available for download from the ROBOT GitHub site<sup id=\"rdp-ebb-cite_ref-GitHubROBOT_16-1\" class=\"reference\"><a href=\"#cite_note-GitHubROBOT-16\">[16]<\/a><\/sup>, along with platform-specific scripts for using ROBOT from the command line. Installation instructions and documentation are available from the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/robot.obolibrary.org\/\" target=\"_blank\">OBO Library<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Architecture\">Architecture<\/span><\/h3>\n<p>We previously described the basic architecture of the tool<sup id=\"rdp-ebb-cite_ref-OvertonROBOT15_17-0\" class=\"reference\"><a href=\"#cite_note-OvertonROBOT15-17\">[17]<\/a><\/sup>, which we summarize here.\n<\/p><p>The ROBOT source code consists of two parts: \u2018robot-core\u2019 and \u2018robot-command.\u2019 \u2018robot-core\u2019 is a library supporting common ontology development tasks, which we call \u201coperations.\u201d \u2018robot-command\u2019 provides a command-line interface divided into \u201ccommands,\u201d each of which wraps a \u2018robot-core\u2019 operation.\n<\/p><p>Most ROBOT operations package low-level functionality provided by OWL API into high-level functionality common to ontology development workflows in the OBO community. For best compatibility, we aim to match the exact version of OWL API used by ROBOT with the exact version used by the latest Prot\u00e9g\u00e9 release. Some operations use Apache Jena.<sup id=\"rdp-ebb-cite_ref-CarrollJena04_18-0\" class=\"reference\"><a href=\"#cite_note-CarrollJena04-18\">[18]<\/a><\/sup> Each operation works with Java objects that represent OWL ontologies, OWL reasoners, OWL classes, etc., while each command works with command-line option strings and files. The commands also perform various conversion and validation steps. The command-line interface uses the Apache Commons CLI library<sup id=\"rdp-ebb-cite_ref-ApacheCommons_19-0\" class=\"reference\"><a href=\"#cite_note-ApacheCommons-19\">[19]<\/a><\/sup> for parsing commands.\n<\/p><p>Each operation has a set of unit tests, built with JUnit<sup id=\"rdp-ebb-cite_ref-JUnit_20-0\" class=\"reference\"><a href=\"#cite_note-JUnit-20\">[20]<\/a><\/sup>, that are executed each time the final product (the JAR file) is generated. Each command in ROBOT is documented in its own web page (e.g., <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/robot.obolibrary.org\/reason\" target=\"_blank\">http:\/\/robot.obolibrary.org\/reason<\/a>). The web pages are authored in Markdown format and contain embedded command-line examples that are parsed and executed as part of our integration tests, with the results compared against a \u201cgold standard\u201d set of outputs. ROBOT\u2019s \u2018diff\u2019 functionality is used when comparing ontology files, otherwise standard file comparison is used. This helps ensure correctness and consistency of documentation and code. The unit tests and integration tests are executed on any pull request onto the codebase via Travis continuous integration (Travis CI)<sup id=\"rdp-ebb-cite_ref-TravisCI_21-0\" class=\"reference\"><a href=\"#cite_note-TravisCI-21\">[21]<\/a><\/sup> so that contributions to the codebase are verified.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Commands_and_operations\">Commands and operations<\/span><\/h3>\n<p>ROBOT currently provides 15 operations (in the \u2018robot-core\u2019 library) and 19 commands (for the command-line interface). Some commands are quite specialized, and most ontology projects will not make use of all of them. Here we provide an overview of the most common and general commands. In each case, the core functionality is supported by operations in the \u2018robot-core\u2019 library, that can be used independently of the command-line interface from any programming language that runs on the JVM.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Conversion\">Conversion<\/span><\/h4>\n<p>A variety of OWL ontology formats are supported, including RDF\/XML, Turtle, Manchester, OBO format, and more. To enable further interoperability, ROBOT includes a <tt>convert<\/tt> command to change between supported ontology formats. A complete list of supported formats can be found in the <tt>convert<\/tt> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/robot.obolibrary.org\/convert\" target=\"_blank\">documentation<\/a>.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Reasoning\">Reasoning<\/span><\/h4>\n<p>Reasoning is one of the most important operations in ROBOT. The <tt>reason<\/tt> command covers two uses: logical validation of an ontology and automatic classification. In both cases, users can choose their preferred reasoner, which is used to perform inference. Large ontologies such as the Gene Ontology typically use ELK<sup id=\"rdp-ebb-cite_ref-KazakovTheIncr13_22-0\" class=\"reference\"><a href=\"#cite_note-KazakovTheIncr13-22\">[22]<\/a><\/sup>, which performs reasoning quickly using the OWL EL profile. Smaller ontologies with richer axiomatization, such as the Relations Ontology, typically use a complete OWL DL reasoner such as HermiT.<sup id=\"rdp-ebb-cite_ref-ShearerHermit08_23-0\" class=\"reference\"><a href=\"#cite_note-ShearerHermit08-23\">[23]<\/a><\/sup>\n<\/p><p>When the <tt>reason<\/tt> command is invoked on an input ontology, ROBOT will initiate a reasoner using the OWL API Reasoner interface. The resulting inferences are checked to ensure the ontology is logically coherent: the ontology must be consistent and have no unsatisfiable classes (i.e., classes that cannot be instantiated without introducing an inconsistency). If the ontology is incoherent, this is reported and execution halts. ROBOT can optionally perform additional checks, such as ensuring that no two classes are inferred to be equivalent post-reasoning.\n<\/p><p>If the ontology is consistent, ROBOT will perform automatic classification. All direct inferred \u2018subClassOf\u2019 axioms are added to the ontology. Generation of other types of axioms can be configured.\n<\/p><p>The assertion of all inferred axioms is often a fundamental step in the release process for biomedical ontologies. Many of these ontology classes only assert a single named superclass (\u2018A subClassOf B\u2019, where B is another class in the ontology), and zero or more anonymous superclasses and\/or anonymous equivalent classes (\u2018A subClassOf\/equivalentTo (R some B)\u2019, where R is an object property). These anonymous classes allow the reasoner to make inferences, which are then asserted. Therefore, in the release version of an ontology, a class may have more than one named superclass.\n<\/p><p>The <tt>reason<\/tt> command has additional \u201chelper\u201d commands. The <tt>relax<\/tt> command asserts entailed subClassOf axioms according to a simple structural rule: an expression \u2018A equivalentTo (R some B) and \u2026\u2019 entails \u2018A subClassOf R some B\u2019. This can be useful as consumers of bio-ontologies often expect to navigate these expressions, e.g., partonomy in GO and Uberon. The <tt>relax<\/tt> command relieves the ontology developer from the need to assert these in addition to the equivalence axioms, and as such it is also often included in release workflows. Finally, the <tt>reduce<\/tt> command removes redundant \u2018subClassOf\u2019 axioms, and can be used after <tt>relax<\/tt> to remove duplicate axioms that were asserted in that step.\n<\/p><p>The <tt>materialize<\/tt> command uses the Expression Materializing Reasoner (EMR) extension to assert inferred expressions of the form \u2018A subClassOf R some B\u2019.<sup id=\"rdp-ebb-cite_ref-GitHubEMR_24-0\" class=\"reference\"><a href=\"#cite_note-GitHubEMR-24\">[24]<\/a><\/sup> Where the <tt>reason<\/tt> command asserts inferred named superclasses, <tt>materialize<\/tt> asserts anonymous superclasses. This is not part of the standard release cycle but can be beneficial for creating complete ontology subsets.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Working_with_external_ontologies\">Working with external ontologies<\/span><\/h3>\n<p>The OBO Foundry aims to coordinate ontologies in a modular fashion, such that parts of some ontologies can be used as building blocks for other ontologies. For example, the ChEBI chemical entities ontology<sup id=\"rdp-ebb-cite_ref-HastingsTheChe13_25-0\" class=\"reference\"><a href=\"#cite_note-HastingsTheChe13-25\">[25]<\/a><\/sup> is used to construct OWL definitions for metabolic processes and activities in the Gene Ontology.<sup id=\"rdp-ebb-cite_ref-HillDove13_26-0\" class=\"reference\"><a href=\"#cite_note-HillDove13-26\">[26]<\/a><\/sup> There are a variety of different strategies for leveraging external ontologies and managing dependencies between ontologies, depending on the use case.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Extractions\">Extractions<\/span><\/h4>\n<p>The <tt>extract<\/tt> command creates a module based on a set of entities to extract (the \u201cseed\u201d). There are four different extraction methods (as specified by the <tt>--method<\/tt> option): MIREOT, TOP, BOT, and STAR.\n<\/p><p>ROBOT\u2019s MIREOT extraction method is based on the principle of the same name<sup id=\"rdp-ebb-cite_ref-CourtotMIREOT11_8-1\" class=\"reference\"><a href=\"#cite_note-CourtotMIREOT11-8\">[8]<\/a><\/sup> and requires that one or more \u201cbottom\u201d entities are specified. Optionally, one or more \u201ctop\u201d entities can also be specified. The command extracts all the \u201cbottom\u201d level entities and their ancestors up to the \u201ctop\u201d level from the input ontology. If no \u201ctop\u201d entities are provided, ancestors up to the top-level entity (\u2018owl: Thing\u2019) are included.\n<\/p><p>The TOP, BOT, and STAR methods make use of the OWL API Syntactic Locality Module Extraction (SLME) implementation, which is guaranteed to capture all <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> logically relevant to the seed set.<sup id=\"rdp-ebb-cite_ref-GrauModular08_27-0\" class=\"reference\"><a href=\"#cite_note-GrauModular08-27\">[27]<\/a><\/sup> The BOT method (\u201cbottom\u201d) includes all relationships between the input entities and their ancestors. The TOP method includes all relationships between the input entities and their descendants. Finally, the STAR method only includes all relationships between input entities. The STAR method produces the smallest outputs, while the TOP method typically produces the largest outputs.\n<\/p><p>In order to support ontology term provenance, the <tt>extract<\/tt> command has an <tt>--annotate-with-source true<\/tt> option that will annotate each extracted term with the URL of the source ontology that it is extracted from.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Removal_and_filtration\">Removal and filtration<\/span><\/h4>\n<p>The <tt>remove<\/tt> and <tt>filter<\/tt> commands are used for fine-grained operations on OWL axioms. <tt>remove<\/tt> allows users to choose which sets of axioms they wish to remove from a target ontology. <tt>filter<\/tt> does the opposite, so that only selected axioms are copied from the input into a new output ontology.\n<\/p><p>These two commands work by starting with a seed set of entities, then applying various selectors to find related entities, and finally selecting which axiom types to remove or filter. We expect only a small number of \u201cpower users\u201d to use this feature directly, but these commands will eventually provide a foundation for other higher-level commands.\n<\/p><p>These commands can be used to generate ontology subsets based on annotations by either filtering for or removing entities with the specified annotation. OBO Foundry ontologies often annotate classes with the \u2018in subset\u2019 property to specify where a class might be used. The annotation selector allows a user to provide a full annotation value or a pattern to match using regular expression.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Merging\">Merging<\/span><\/h4>\n<p>The <tt>merge<\/tt> command combines two or more separate input ontologies into a single ontology. It also provides the ability to merge all imported ontologies of a single input ontology into one main ontology, which is often used when creating a release.\n<\/p><p>Merging imported ontologies (specified by import statements) into the input ontology is performed automatically so that the user does not need to list each imported ontology as an input. We offer the option (<tt>--collapse-import-closure false<\/tt>) to turn this feature off, supporting cases in which users may merge multiple input ontologies that have import statements but want to keep their imports separate.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Querying_and_reporting\">Querying and reporting<\/span><\/h3>\n<p>Ontology workflows typically include query operations over the ontology, producing reports which may be informative for both editors and users of the ontology, e.g., a table of all classes plus their textual definitions. Query operations can also be used for validation checks. The SPARQL query language provides a universal and declarative way for ontology maintainers to create ontology reports and validation checks.<sup id=\"rdp-ebb-cite_ref-HarrisSPARQL13_28-0\" class=\"reference\"><a href=\"#cite_note-HarrisSPARQL13-28\">[28]<\/a><\/sup> ROBOT provides a convenient way to perform queries with the <tt>query<\/tt> command, or validation checks using <tt>verify<\/tt>. Additionally, the <tt>report<\/tt> command includes a configurable package of standard queries for OBO projects that can be used in any ontology workflow, without requiring the maintainer to be familiar with SPARQL.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Querying\">Querying<\/span><\/h4>\n<p>ROBOT\u2019s <tt>query<\/tt> command runs SPARQL queries on ontologies or other RDF resources. This can be used by an ontology maintainer to either perform interactive queries, or more typically to include standard queries into an ontology workflow. The <tt>query<\/tt> command wraps one of the few operations that uses Apache Jena<sup id=\"rdp-ebb-cite_ref-CarrollJena04_18-1\" class=\"reference\"><a href=\"#cite_note-CarrollJena04-18\">[18]<\/a><\/sup>, rather than OWL API. The Jena API allows ROBOT to load an ontology as a collection of triples contained by an RDF Model object. It provides a SPARQL query engine for those models, which we use to run all queries.\n<\/p><p>\u2018SPARQL SELECT\u2019 queries produce a comma- or tab-separated table of results. ASK queries produce a file with a Boolean value. \u2018SPARQL CONSTRUCT\u2019 queries produce an RDF file, which can be further processed by ROBOT or merged back into the loaded ontology. \u2018CONSTRUCT\u2019s provide a convenient way of performing \u201cmacro\u201d style expansion.<sup id=\"rdp-ebb-cite_ref-MungallTaking11_29-0\" class=\"reference\"><a href=\"#cite_note-MungallTaking11-29\">[29]<\/a><\/sup> \u2018SPARQL UPDATE\u2019 queries insert and\/or remove data directly in an ontology (as an RDF Model). ROBOT converts the updated RDF Model back to an OWL API ontology object to be saved in any of the supported syntaxes.\n<\/p><p>The <tt>query<\/tt> command supports an option to load imported ontologies as named graphs with the <tt>--use-graphs<\/tt> option. If this is set to <tt>true<\/tt>, the imports can be queried as named graphs (the name being that ontology\u2019s IRI) and the default graph is a union of all graphs. Using the default graph is similar to conducting a <tt>merge<\/tt> of all the imports prior to querying, but the distinction between imports would be lost in a <tt>merge<\/tt>.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Verification\">Verification<\/span><\/h4>\n<p>The <tt>verify<\/tt> command is a variation on the \u2018SPARQL SELECT\u2019 execution. The queries are used to ensure that an ontology conforms to a predetermined set of conditions, by, for example, ensuring that no class has multiple textual definitions. Given a SELECT query, <tt>verify<\/tt> will succeed (i.e., exit with status code 0) if no results are returned. It will fail (i.e., exit with a non-zero status code) if any results are return from the query. So, given a SPARQL query that SELECTs for invalid data, the <tt>verify<\/tt> command will verify that the ontology (or other resource) does not contain any such invalid data.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Reporting\">Reporting<\/span><\/h4>\n<p>The <tt>report<\/tt> command is an extension of <tt>query<\/tt> and <tt>verify<\/tt> that provides a series of configurable quality control (QC) checks for an ontology and returns a spreadsheet or YAML output of the violations. The spreadsheet is output in either comma- or tab-separated format and easy for a user to read, while the YAML output can be easily parsed by other programs.\n<\/p><p>The QC checks include annotation checks, logical checks, and metadata checks. Annotations are important to facilitate human comprehension, so the <tt>report<\/tt> command finds cases where missing or duplicate annotations could cause problems. Logical checks look at the structural coherency and consistency of the ontology. Finally, <tt>report<\/tt> identifies missing ontology metadata, as specified by OBO Foundry recommendations.\n<\/p><p>There are three levels of violations that are reported: ERROR, WARN, and INFO. An ERROR is the most severe, such as a missing or duplicate label. By default, the <tt>report<\/tt> command fails if there are any ERROR-level violations, halting any automated build processes. These types of violations must be fixed before publishing an ontology. WARN-level violations should be fixed as soon as possible, e.g., inferred one-to-one class equivalencies, which are typically unintended in OBO projects. INFO is for recommended fixes that help maintain consistency across OBO Foundry ontologies, such as beginning a definition with an uppercase letter and ending with a period. <tt>report<\/tt> can be configured with a command-line option to fail on a different violation level or to never fail, regardless of any violations. We document each QC check with a suggestion for a manual fix that the user can apply.\n<\/p><p>A default \u201cprofile\u201d with report levels for each QC check is provided by ROBOT, but users are also able to create their own profiles. In these profiles, they can change the violation levels of individual checks, choose to exclude certain checks, and add their own checks as SPARQL queries. For example, some ontologies may categorize a class lacking a textual definition as an error, while others may categorize this as a warning. One of our goals is to converge on a standard profile that is maximally useful for the set of all ontologies in the OBO library, encouraging adoption of common quality control checks.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Repairs\">Repairs<\/span><\/h4>\n<p>Although most problems raised by <tt>validate<\/tt> and <tt>report<\/tt> must be fixed manually, ROBOT also provides a <tt>repair<\/tt> command that can automatically fix certain problems. The current implementation will merge annotations on duplicate axioms and update references to deprecated classes when they are annotated with a suggested replacement. We intend to extend <tt>repair<\/tt> to a wider range of common problems for which the correct fix is clear.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Templated_ontology_development\">Templated ontology development<\/span><\/h3>\n<p>ROBOT provides a template-driven ontology term generation system. Users also have the option to plug other term generation systems such as Dead Simple OWL Design Patterns (DOS-DPs)<sup id=\"rdp-ebb-cite_ref-Osumi-SutherlandDead17_30-0\" class=\"reference\"><a href=\"#cite_note-Osumi-SutherlandDead17-30\">[30]<\/a><\/sup> into their workflow.\n<\/p><p>A huge amount of data is stored in spreadsheets and databases, and tabular formats are well suited to many sorts of data. ROBOT\u2019s <tt>template<\/tt> command allows users to convert tabular data into RDF\/OWL format. A ROBOT template is simply a tab-separated values (TSV) or comma-separated values (CSV) file with some special conventions, which are outlined in the ROBOT <tt>template<\/tt> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/robot.obolibrary.org\/template\" target=\"_blank\">documentation<\/a>.\n<\/p><p>These templates can be used for modular ontology development. The template spreadsheets may be maintained as part of the ontology\u2019s source code repository, and instead of directly editing the ontology file, developers edit rows in the spreadsheet which correspond to terms in the ontology. The <tt>template<\/tt> command is then used to generate a module of the ontology, which is included as an import statement in the editors\u2019 version of the ontology and merged during the release process.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Workflows\">Workflows<\/span><\/h3>\n<p>A workflow consists of a set of tasks coordinated by some workflow system. Ontology workflows consist of tasks such as executing QC checks, building import modules, reasoning over ontologies, and generating various ontology release products.\n<\/p><p>ROBOT itself is not a workflow manager, although it allows multiple commands to be chained together into one long command. When chaining ROBOT commands, the output ontology from one command is passed directly as the input to the next command. For example, chaining may be used to replace two commands that merge ontologies and then reason over the merged product:\n<\/p>\n<ul><li> <tt>robot merge --input ont-1.owl --input ont-2.owl --output merged.owl<\/tt><\/li>\n<li> <tt>robot reason --input merged.owl --output reasoned.owl<\/tt><\/li><\/ul>\n<p>Instead of creating the merged product and running <tt>reason<\/tt> over that, it can be done in one command:\n<\/p>\n<ul><li> <tt>robot merge --input ont-1.owl --input ont-2.owl reason --output reasoned.owl<\/tt><\/li><\/ul>\n<p>The key advantage to chaining is that ontologies do not have to be serialized and parsed between each step; the same OWL API ontology object is maintained in memory and passed through the chain of ROBOT commands. For large ontologies, chaining can vastly improve ROBOT\u2019s performance.\n<\/p><p>Because ROBOT commands can be executed on the command line, a number of different workflow systems can be used. We highlight the use of GNU Make<sup id=\"rdp-ebb-cite_ref-GNUMake_31-0\" class=\"reference\"><a href=\"#cite_note-GNUMake-31\">[31]<\/a><\/sup>, which is typically used to compile software. A Makefile consists of a set of rules used to make \u201ctargets.\u201d In ontology development, the Makefile is used for automated tasks such as preparing an ontology for release. In this case, the targets are usually ontology files. The \u201crecipes\u201d for the rules are Unix-style system commands, carried out by the <tt>make<\/tt> command.\n<\/p><p>ROBOT commands can be used as the \u201crecipes\u201d to make the \u201ctargets.\u201d A typical workflow will not use all 19 of the ROBOT commands. For example, not all ontology projects may use ROBOT templates and therefore not all release workflows need to include the <tt>template<\/tt> command. Ontology developers can decide which commands are needed to perform the release and build a workflow around those commands. Figure 1 shows a standard way in which a selection of ROBOT commands is combined for a release workflow.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Jackson_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"a4b180f8247228a47a8dd2447738bea2\"><img alt=\"Fig1 Jackson BMCBioinformatics2019 20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/83\/Fig1_Jackson_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> The ROBOT release workflow. A typical release workflow using ROBOT. The ontology edit file ONT-edit.owl is first verified as a quality control check with ROBOT <tt>verify<\/tt>. Then, text files containing lists of external ontology terms in the imports directory are used to regenerate import modules using <tt>extract<\/tt>, ensuring that the imports are up-to-date. ONT-edit.owl is then passed through a series of ROBOT commands (<tt>reason<\/tt>, <tt>relax<\/tt>, <tt>reduce<\/tt>, and <tt>annotate<\/tt>) to generate the release, ONT.owl. Finally, ONT.owl is converted to OBO format.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>First, quality control checks are run over the editors\u2019 version of the ontology with <tt>report<\/tt> or <tt>verify<\/tt>. These look for equivalent classes, trailing whitespace in annotations, self-references, incorrect cross-reference syntax, and missing labels. The results are saved to a specified \"reports\/\" directory. If there are any ERROR-level violations, the task will fail and write the violations to a table so that they can be easily identified. This step allows developers to quickly see if new changes have introduced any problems within the ontology and fix them before releasing.\n<\/p><p>Assuming the initial QC check step has completed successfully, the next step is to create the import modules. The ROBOT <tt>extract<\/tt> is run for each entry in a list of imports, which have corresponding term files (for the seed set) in the \"imports\/\" directory. This creates all the import modules in the same \"imports\/\" directory. This ensures that when an ontology is released with external terms, all external terms are up-to-date with the released versions of the source ontologies. Releasing out-of-date external terms can cause confusion, as the term will show both the old and new details in ontology search services like Ontobee<sup id=\"rdp-ebb-cite_ref-XiangOntobee11_5-1\" class=\"reference\"><a href=\"#cite_note-XiangOntobee11-5\">[5]<\/a><\/sup> and the Ontology Lookup Service.<sup id=\"rdp-ebb-cite_ref-C.C3.B4t.C3.A9TheOnt10_32-0\" class=\"reference\"><a href=\"#cite_note-C.C3.B4t.C3.A9TheOnt10-32\">[32]<\/a><\/sup> Additional QC checks can be run over the full ontology with imports using the <tt>verify<\/tt> command or by running <tt>report<\/tt> again.\n<\/p><p>Last, the main release products are created: the OWL file and the OBO file. To create the OWL release, the editors\u2019 file is passed through a series of chained ROBOT commands: <tt>reason<\/tt>, <tt>relax<\/tt>, <tt>reduce<\/tt>, and <tt>annotate<\/tt>. This series of commands helps to ensure that the released ontology is both easy to browse and understand, as well as free of any redundant axioms. If any of these commands fail, the Make process will terminate with the corresponding error message. For example, if an ontology is incoherent the <tt>reason<\/tt> step will fail. Finally, the <\/tt>annotate<\/tt> command adds the version IRI to the ontology metadata. This OWL file is then converted to OBO format, at which point all targets are copied to a dated release directory.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_Ontology_Development_Kit\">The Ontology Development Kit<\/span><\/h3>\n<p>Creating a Makefile to coordinate all these steps can be challenging. We make this easier for ontology developers by providing an Ontology Development Kit (ODK).<sup id=\"rdp-ebb-cite_ref-GitHubODK_33-0\" class=\"reference\"><a href=\"#cite_note-GitHubODK-33\">[33]<\/a><\/sup> This can be used to create a GitHub repository following a standard layout, with a standard Makefile following the workflow detailed above. The resulting GitHub repository will also be automatically configured to run the validation steps (such as <tt>report<\/tt>) of the workflow via Travis CI.<sup id=\"rdp-ebb-cite_ref-TravisCI_21-1\" class=\"reference\"><a href=\"#cite_note-TravisCI-21\">[21]<\/a><\/sup> The workflow can also be executed using Docker with ODK containers released on DockerHub.<sup id=\"rdp-ebb-cite_ref-DHObolibrary_34-0\" class=\"reference\"><a href=\"#cite_note-DHObolibrary-34\">[34]<\/a><\/sup> This allows easy execution of workflows on either the local computer of an ontology developer, with Travis CI, or through scalable-build tools such as Jenkins.<sup id=\"rdp-ebb-cite_ref-GitHubODK_33-1\" class=\"reference\"><a href=\"#cite_note-GitHubODK-33\">[33]<\/a><\/sup>\n<\/p><p>ODK builds on ROBOT and demonstrates ROBOT\u2019s utility, but a full discussion is beyond the scope of this article.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<p>While there are many other Semantic Web and OWL development tools available, a number of factors has driven the OBO community to build and support ROBOT. First, the OBO commitment to open-source development is a strong reason to use open-source software. Second, our wide reliance on the free and open Prot\u00e9g\u00e9 editor is a strong reason to use the same OWL API library that it is built upon. Third, there is a strong incentive to pool our limited resources and invest in shared tooling. Fourth, the OBO community has a range of conventions that support interoperability, and our workflows are simpler if we build these assumptions into the tools. Points three and four are clearly in tension: what is the right balance to strike between reusing general Semantic Web tools and building our own? Prot\u00e9g\u00e9, OWL API, and various OWL reasoners are general tools that we use, for instance, but we have strong conventions in our community for identifiers, release artifacts, <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a>, quality control, etc. that these tools do not help us to enforce. Our compromise is to reuse open-source tools as much as practical and invest community resources in customizing general tools such as OWL API to serve the needs of our community. A growing number of ontology developers are using ROBOT to help automate their quality-checking and release workflows. Two case studies are described here.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Ontology_for_Biomedical_Investigations\">Ontology for Biomedical Investigations<\/span><\/h3>\n<p>The Ontology for Biomedical Investigations (OBI) is an OBO Foundry ontology that aims to describe the processes, agents, devices, inputs, and outputs of scientific investigations.<sup id=\"rdp-ebb-cite_ref-BandrowskiTheOnt16_35-0\" class=\"reference\"><a href=\"#cite_note-BandrowskiTheOnt16-35\">[35]<\/a><\/sup> When the project began more than a decade ago, development was done in Prot\u00e9g\u00e9, without any automation, and hosted on SourceForge. Today, OBI uses ROBOT to implement an automated workflow, supported by GitHub pull requests and Travis CI testing. More than 50 people have contributed to OBI development, including two of the authors of this paper (Overton and Jackson).\n<\/p><p>OBI has always imported a range of terms from other OBO projects, and OBI developers have maintained a number of separate OWL files to facilitate concurrent development by different groups of developers. When it comes time to prepare a new release of OBI, the various OWL files must be merged, tested, and reasoned over.\n<\/p><p>In the early days of the project, OBI developer Alan Ruttenberg wrote a series of scripts for quality control and common operations, but merging, reasoning, and testing a new release still involved many hours of work by OBI developers. In 2013, James Overton developed a precursor to ROBOT: an automated build tool written in Java, using OWL API and Apache Ant, that automated some of the build, test, and release processes. This drastically reduced the time required to make a release, allowing for more frequent releases. While this code was specific to OBI workflows, some of it was used in early versions of ROBOT.\n<\/p><p>In 2017, OBI moved from SourceForge to GitHub, and the release workflow was updated to use ROBOT throughout. This change vastly increased the degree of automation for ontology development tasks, expanded capabilities, and allowed OBI to pool some of its development resources with the wider OBO community to support shared tooling. OBI currently uses <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/obi-ontology\/obi\/blob\/5fc065729fc7eb013dafd14690559621b5606057\/Makefile\" target=\"_blank\">a Makefile<\/a> that defines a range of tasks for managing imports, converting templates, merging, reasoning, testing, and releasing new versions of OBI. The Makefile specifies various target files, and most target files are generated from a single ROBOT command or a single chain of ROBOT commands. The key steps are:\n<\/p>\n<ol><li> Update imports from upstream ontologies (currently using Ontofox<sup id=\"rdp-ebb-cite_ref-XiangOntoFox10_6-1\" class=\"reference\"><a href=\"#cite_note-XiangOntoFox10-6\">[6]<\/a><\/sup>). OBI imports subsets of terms from more than a dozen OBO projects. As discussed, ROBOT supports this functionality with <tt>extract<\/tt>, but OBI\u2019s use of Ontofox predates ROBOT development and has not yet been migrated.<\/li>\n<li> Normalize RDF\/XML for cleaner history of changes in the version control system (\u2018robot convert\u2019). Different versions of OWL API have slightly different serialization behavior, which can lead to spurious reports of differences that make it more difficult to see relevant changes to the source files.<\/li>\n<li> Convert template files (TSV) to OWL modules (\u2018robot template\u2019). Templates often make it easier for domain experts to contribute to ontology development and enforce ontology design patterns that improve the overall quality of OBI.<\/li>\n<li> Merge imports and templates with \u2018obi-edit.owl\u2019 (\u2018robot merge\u2019). OBI uses a number of import and template files to enforce a separation of concerns, rather than making all changes in a single source file. These are merged into a single release file.<\/li>\n<li> Use \u2018SPARQL CONSTRUCT\u2019 queries to update various term annotations (\u2018robot query\u2019). Some standard term metadata can be automatically added and updated, rather than manually maintained.<\/li>\n<li> Run an automated test suite (\u2018robot verify\u2019). A range of quality control checks helps to ensure that errors have not been introduced into OBI by recent changes.<\/li>\n<li> Run the HermiT reasoner (\u2018robot reason\u2019). Reasoning checks the logical consistency of OBI and performs automated classification of terms.<\/li>\n<li> Update annotations for release (\u2018robot annotate\u2019). These annotations include the dated version IRI of this release of OBI.<\/li>\n<li> Extract the OBI Core subset (\u2018robot extract\u2019). The OBI Core subset provides approximately 100 important terms for educational and documentation purposes.<\/li>\n<li> Create a list of OBI terms (\u2018robot query\u2019). The term list is used to report on the new terms added to OBI with each release.<\/li><\/ol>\n<h3><span class=\"mw-headline\" id=\"Disease_Ontology\">Disease Ontology<\/span><\/h3>\n<p>The Disease Ontology (DO) is an OBO Foundry ontology that provides a standardized description of human diseases, including the phenotypic characteristics, symptoms, genetic bases, and related medical terminology. These terms are used by various model organism databases to provide a consistent representation of diseases.<sup id=\"rdp-ebb-cite_ref-BelloDisease18_36-0\" class=\"reference\"><a href=\"#cite_note-BelloDisease18-36\">[36]<\/a><\/sup> The DO is developed at the University of Maryland School of Medicine by Lynn Schriml and her team, which currently includes one of the authors (Jackson).\n<\/p><p>In order to accurately and thoroughly describe the different aspects of diseases, DO makes use of more than 10 other biological ontologies. In the past, all DO imports were manually created and maintained. This led to inconsistencies as ontologies were updated and expanded, and also made it very difficult to add new entities to the imports.\n<\/p><p>In 2018, DO switched their entire automated build process (contained in the Makefile) from OWLTools<sup id=\"rdp-ebb-cite_ref-GitHubOwlTools_14-1\" class=\"reference\"><a href=\"#cite_note-GitHubOwlTools-14\">[14]<\/a><\/sup> and OORT<sup id=\"rdp-ebb-cite_ref-GitHubOwlTools-Oort_15-1\" class=\"reference\"><a href=\"#cite_note-GitHubOwlTools-Oort-15\">[15]<\/a><\/sup> to ROBOT. Instead of manually updating import modules, all required entities are now specified in text files. When a developer wishes to add a new imported entity, they simply add a line to the text file and run <tt>make imports<\/tt>. All imports are automatically regenerated during releases, as well, to keep the information up-to-date.\n<\/p><p>Before ROBOT, the monthly DO releases took multiple hours to run and required additional hours of manual editing and review. Now, DO developers simply run the <tt>make release<\/tt> command and all content is generated in less than 20\u2009minutes. The release process makes use of ROBOT commands such as <tt>report<\/tt> to run quality control checks over the release products and reduce the time spent reviewing output.\n<\/p><p>Both ROBOT <tt>verify<\/tt> and <\/tt>report<\/tt> are used for DO\u2019s Travis CI system.<sup id=\"rdp-ebb-cite_ref-TravisCI_21-2\" class=\"reference\"><a href=\"#cite_note-TravisCI-21\">[21]<\/a><\/sup> Each time a new commit is pushed to the GitHub repository, a series of queries is run against the ontology files to ensure they meet certain standards. If they do not (or if <tt>report<\/tt> fails with an ERROR-level violation), Travis CI notifies developers that the checks have failed with a red \u201cX\u201d next to the commit. Clicking on the red \u201cX\u201d will take the user to Travis CI, where they can see the command log to determine what caused the failure. On success, a green check mark is displayed next to the commit to show that the checks have passed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>ROBOT makes it easy for ontology developers to automate a wide range of tedious and error-prone development tasks, freeing their time to focus on other important parts of the ontology life cycle. Circa 2010, most OBO projects were manually edited, with manual imports, manually tested, and manually released using only Prot\u00e9g\u00e9. With ROBOT, ontology developers have a convenient tool for building powerful workflows that include format conversion, reasoning, extracting, querying, updating, testing, reporting, templating, and more. Low-level OWL API and Apache Jena operations are packaged into a library of high-level operations, and these operations are wrapped in a convenient command-line interface that is supported on the common server and desktop platforms. With ODK, developers benefit not only from ROBOT, but additionally from a wide range of best practices and standard procedures developed and shared by the OBO community.\n<\/p><p>ROBOT is open-source software developed by a community of a dozen contributors with more than 1000 commits, hundreds of closed issues, and six releases. The ROBOT source code is freely available on GitHub and Maven Central. Documentation for the library is available on Javadoc.io, and documentation for the commands is available on <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/robot.obolibrary.org\" target=\"_blank\">our website<\/a>, where you will also find examples of usage, test files, and explanations of common errors.\n<\/p><p>With ROBOT, we have tried to strike a balance between general tools such as Prot\u00e9g\u00e9 and the specific needs of the OBO community, and to deliver the benefits of automation from software development to ontology development.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Software_availability_and_requirements\">Software availability and requirements<\/span><\/h2>\n<p><b>Project name<\/b>: ROBOT (ROBOT is an OBO Tool)\n<\/p><p><b>Project home page<\/b>: <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/robot.obolibrary.org\/\" target=\"_blank\">http:\/\/robot.obolibrary.org\/<\/a>\n<\/p><p><b>Operating system(s)<\/b>: Platform-independent\n<\/p><p><b>Programming language<\/b>: Java 8\n<\/p><p><b>Other requirements<\/b>: None for the command-line tool. The ROBOT library depends on the following: Apache Jena, SnakeYAML, OpenCSV, FasterXML Jackson, OWL API, Apache Commons IO, Apache Maven, JSONLD-JAVA, Prot\u00e9g\u00e9, JUnit, SLF4J, and fmt-maven-plugin.\n<\/p><p><b>License<\/b>: ROBOT is available under BSD 3. Dependencies are available under Apache 2.0 (Jena, SnakeYAML, OpenCSV, Jackson, OWL API, Commons IO, and Maven), BSD 3 (JSONLD-JAVA and Prot\u00e9g\u00e9), EPL-1.0 (JUnit), and the MIT License (SLF4J and fmt-maven-plugin).\n<\/p><p><b>Any restrictions to use by non-academics<\/b>: None other than those specified by the licenses.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>CSV<\/b>: comma-separated values\n<\/p><p><b>DL<\/b>: description logic\n<\/p><p><b>DO<\/b>: Disease Ontology\n<\/p><p><b>DOS-DP<\/b>: Dead Simple OWL Design Patterns\n<\/p><p><b>EMR<\/b>: Expression Materializing Reasoner\n<\/p><p><b>GUI<\/b>: graphical user interface\n<\/p><p><b>OBI<\/b>: Ontology for Biomedical Investigations\n<\/p><p><b>OBO<\/b>: Open Biological Ontologies\n<\/p><p><b>ODK<\/b>: Ontology Development Kit\n<\/p><p><b>OWL<\/b>: Web Ontology Language\n<\/p><p><b>ROBOT<\/b>: ROBOT is an OBO Tool\n<\/p><p><b>TSV<\/b>: tab-separated values\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We thank David Osumi-Sutherland for requirements and testing and Nico Matentzoglu for submitting bug reports and feature requests. We also thank all of our users and everyone who has participated in ROBOT development, from contributing code and documentation, to reporting bugs and requesting features.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Contributions\">Contributions<\/span><\/h3>\n<p>RCJ, JAO and JPB wrote the manuscript. NH edited the manuscript. RCJ, JPB, ED, JAO and CJM developed and implemented the software. All authors read and approved the final manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work and related resources are supported by the following grants:\n<\/p>\n<ul><li> From the Department of Health and Human Services (NIH), for \u201cImmune Epitope Database and Analysis Resource Program\u201d (HHSN272201200010C)<\/li>\n<li> From the National Institute of Allergy and Infectious Diseases (NIH), for \u201cHuman immune signatures of Dengue virus and Mycobacterium Tuberculosis exposure in infection, disease and vaccination\u201d (1-U19-AI-118626-01)<\/li>\n<li> From the National Human Genome Research Institute (NIH), for \u201cServices to support the OBO foundry standards\u201d (R24-HG010032)<\/li>\n<li> From the National Human Genome Research Institute (NIH), for \u201cGene Ontology Consortium\u201d (U41HG002273)<\/li><\/ul>\n<p>These funding bodies did not play any role in the design of ROBOT, the interpretation of data, or the writing of this manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Availability_of_data_and_materials\">Availability of data and materials<\/span><\/h3>\n<p>All code is available from <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/ontodev\/robot\/\" target=\"_blank\">https:\/\/github.com\/ontodev\/robot\/<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare that they have no competing interests. ROBOT development is largely funded by the grants acknowledged above. Knocean, Inc. is subcontracted for ROBOT development from those grants. Overton is president and owner of Knocean, Inc. Jackson is a contractor for Knocean, Inc.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-NoyTheOnt10-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NoyTheOnt10_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Noy, N.; Tudorache, T.; Nyulas, C. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3041389\" target=\"_blank\">\"The ontology life cycle: Integrated tools for editing, publishing, peer review, and evolution of ontologies\"<\/a>. <i>AMIA Annual Symposium Proceedings<\/i> <b>2010<\/b>: 552\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3041389\/\" target=\"_blank\">PMC3041389<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21347039\" target=\"_blank\">21347039<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3041389\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3041389<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+ontology+life+cycle%3A+Integrated+tools+for+editing%2C+publishing%2C+peer+review%2C+and+evolution+of+ontologies&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Noy%2C+N.%3B+Tudorache%2C+T.%3B+Nyulas%2C+C.+et+al.&rft.au=Noy%2C+N.%3B+Tudorache%2C+T.%3B+Nyulas%2C+C.+et+al.&rft.date=2010&rft.volume=2010&rft.pages=552%E2%80%936&rft_id=info:pmc\/PMC3041389&rft_id=info:pmid\/21347039&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3041389&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HorridgeSupport06-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HorridgeSupport06_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Horridge, M.; Tsarkov, D. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/webont.org\/owled\/2006\/accepted06.html\" target=\"_blank\">\"Supporting early adoption of OWL 1.1 with Prot\u00e9g\u00e9-OWL and FaCT++\"<\/a>. <i>OWL: Experiences and Directions 2006<\/i>: 1\u20137<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/webont.org\/owled\/2006\/accepted06.html\" target=\"_blank\">http:\/\/webont.org\/owled\/2006\/accepted06.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Supporting+early+adoption+of+OWL+1.1+with+Prot%C3%A9g%C3%A9-OWL+and+FaCT%2B%2B&rft.jtitle=OWL%3A+Experiences+and+Directions+2006&rft.aulast=Horridge%2C+M.%3B+Tsarkov%2C+D.&rft.au=Horridge%2C+M.%3B+Tsarkov%2C+D.&rft.date=2006&rft.pages=1%E2%80%937&rft_id=http%3A%2F%2Fwebont.org%2Fowled%2F2006%2Faccepted06.html&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmithTheOBO07-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmithTheOBO07_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smith, B.; Ashburner, M.; Rosse, C. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2814061\" target=\"_blank\">\"The OBO Foundry: Coordinated evolution of ontologies to support biomedical data integration\"<\/a>. <i>Nature Biotechnology<\/i> <b>25<\/b> (11): 1251\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt1346\" target=\"_blank\">10.1038\/nbt1346<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2814061\/\" target=\"_blank\">PMC2814061<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17989687\" target=\"_blank\">17989687<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2814061\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2814061<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+OBO+Foundry%3A+Coordinated+evolution+of+ontologies+to+support+biomedical+data+integration&rft.jtitle=Nature+Biotechnology&rft.aulast=Smith%2C+B.%3B+Ashburner%2C+M.%3B+Rosse%2C+C.+et+al.&rft.au=Smith%2C+B.%3B+Ashburner%2C+M.%3B+Rosse%2C+C.+et+al.&rft.date=2007&rft.volume=25&rft.issue=11&rft.pages=1251%E2%80%935&rft_id=info:doi\/10.1038%2Fnbt1346&rft_id=info:pmc\/PMC2814061&rft_id=info:pmid\/17989687&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2814061&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeOnto-Animal15-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeOnto-Animal15_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">He, Y.; Zheng, J.; Lin, Y. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ceur-ws.org\/Vol-1515\/\" target=\"_blank\">\"Onto-animal tools for reusing ontologies, generating and editing ontology terms, and dereferencing ontology terms\"<\/a>. <i>Proceedings of the 2015 International Conference on Biomedical Ontology<\/i>: 1\u20132<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ceur-ws.org\/Vol-1515\/\" target=\"_blank\">http:\/\/ceur-ws.org\/Vol-1515\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Onto-animal+tools+for+reusing+ontologies%2C+generating+and+editing+ontology+terms%2C+and+dereferencing+ontology+terms&rft.jtitle=Proceedings+of+the+2015+International+Conference+on+Biomedical+Ontology&rft.aulast=He%2C+Y.%3B+Zheng%2C+J.%3B+Lin%2C+Y.&rft.au=He%2C+Y.%3B+Zheng%2C+J.%3B+Lin%2C+Y.&rft.date=2015&rft.pages=1%E2%80%932&rft_id=http%3A%2F%2Fceur-ws.org%2FVol-1515%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XiangOntobee11-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-XiangOntobee11_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-XiangOntobee11_5-1\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xiang, Z.; Mungall, C.; Ruttenberg, A. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ceur-ws.org\/Vol-833\/\" target=\"_blank\">\"Ontobee: A Linked Data Server and Browser for Ontology Terms\"<\/a>. <i>Proceedings of the Second International Conference on Biomedical Ontology<\/i>: 279\u201381<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ceur-ws.org\/Vol-833\/\" target=\"_blank\">http:\/\/ceur-ws.org\/Vol-833\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ontobee%3A+A+Linked+Data+Server+and+Browser+for+Ontology+Terms&rft.jtitle=Proceedings+of+the+Second+International+Conference+on+Biomedical+Ontology&rft.aulast=Xiang%2C+Z.%3B+Mungall%2C+C.%3B+Ruttenberg%2C+A.+et+al.&rft.au=Xiang%2C+Z.%3B+Mungall%2C+C.%3B+Ruttenberg%2C+A.+et+al.&rft.date=2012&rft.pages=279%E2%80%9381&rft_id=http%3A%2F%2Fceur-ws.org%2FVol-833%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XiangOntoFox10-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-XiangOntoFox10_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-XiangOntoFox10_6-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xiang, Z.; Courtot, M.; Brinkman, R.R. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2911465\" target=\"_blank\">\"OntoFox: Web-based support for ontology reuse\"<\/a>. <i>BMC Research Notes<\/i> <b>3<\/b>: 175. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1756-0500-3-175\" target=\"_blank\">10.1186\/1756-0500-3-175<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2911465\/\" target=\"_blank\">PMC2911465<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20569493\" target=\"_blank\">20569493<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2911465\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2911465<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=OntoFox%3A+Web-based+support+for+ontology+reuse&rft.jtitle=BMC+Research+Notes&rft.aulast=Xiang%2C+Z.%3B+Courtot%2C+M.%3B+Brinkman%2C+R.R.+et+al.&rft.au=Xiang%2C+Z.%3B+Courtot%2C+M.%3B+Brinkman%2C+R.R.+et+al.&rft.date=2010&rft.volume=3&rft.pages=175&rft_id=info:doi\/10.1186%2F1756-0500-3-175&rft_id=info:pmc\/PMC2911465&rft_id=info:pmid\/20569493&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2911465&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XiangOntoRat15-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-XiangOntoRat15_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xiang, Z.; Zheng, J.; Lin. Y. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4362828\" target=\"_blank\">\"Ontorat: Automatic generation of new ontology terms, annotations, and axioms based on ontology design patterns\"<\/a>. <i>Journal of Biomedical Semantics<\/i> <b>6<\/b>: 4. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F2041-1480-6-4\" target=\"_blank\">10.1186\/2041-1480-6-4<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4362828\/\" target=\"_blank\">PMC4362828<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25785185\" target=\"_blank\">25785185<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4362828\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4362828<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ontorat%3A+Automatic+generation+of+new+ontology+terms%2C+annotations%2C+and+axioms+based+on+ontology+design+patterns&rft.jtitle=Journal+of+Biomedical+Semantics&rft.aulast=Xiang%2C+Z.%3B+Zheng%2C+J.%3B+Lin.+Y.+et+al.&rft.au=Xiang%2C+Z.%3B+Zheng%2C+J.%3B+Lin.+Y.+et+al.&rft.date=2015&rft.volume=6&rft.pages=4&rft_id=info:doi\/10.1186%2F2041-1480-6-4&rft_id=info:pmc\/PMC4362828&rft_id=info:pmid\/25785185&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4362828&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CourtotMIREOT11-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CourtotMIREOT11_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-CourtotMIREOT11_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Courtot, M.; Gibson, F.; Lister, A. L. et al. (2011). \"MIREOT: The minimum information to reference an external ontology term\". <i>Applied Ontology<\/i> <b>6<\/b> (1): 23\u201333. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2FAO-2011-0087\" target=\"_blank\">10.3233\/AO-2011-0087<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MIREOT%3A+The+minimum+information+to+reference+an+external+ontology+term&rft.jtitle=Applied+Ontology&rft.aulast=Courtot%2C+M.%3B+Gibson%2C+F.%3B+Lister%2C+A.+L.+et+al.&rft.au=Courtot%2C+M.%3B+Gibson%2C+F.%3B+Lister%2C+A.+L.+et+al.&rft.date=2011&rft.volume=6&rft.issue=1&rft.pages=23%E2%80%9333&rft_id=info:doi\/10.3233%2FAO-2011-0087&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeTheExt18-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeTheExt18_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">He, Y.; Xiang, Z.; Zheng, J. et al. (2018). \"The eXtensible ontology development (XOD) principles and tool implementation to support ontology interoperabilitye\". <i>Journal of Biomedical Semantics<\/i> <b>9<\/b>: 3. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13326-017-0169-2\" target=\"_blank\">10.1186\/s13326-017-0169-2<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+eXtensible+ontology+development+%28XOD%29+principles+and+tool+implementation+to+support+ontology+interoperabilitye&rft.jtitle=Journal+of+Biomedical+Semantics&rft.aulast=He%2C+Y.%3B+Xiang%2C+Z.%3B+Zheng%2C+J.+et+al.&rft.au=He%2C+Y.%3B+Xiang%2C+Z.%3B+Zheng%2C+J.+et+al.&rft.date=2018&rft.volume=9&rft.pages=3&rft_id=info:doi\/10.1186%2Fs13326-017-0169-2&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HorridgeIgnit07-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HorridgeIgnit07_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Horridge, M.; Bechhofer, S.; Noppens, O. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/webont.org\/owled\/2007\/Proceedings.html\" target=\"_blank\">\"Igniting the OWL 1.1 Touch Paper: The OWL API\"<\/a>. <i>OWL: Experiences and Directions 2007<\/i>: 1\u20139<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/webont.org\/owled\/2007\/Proceedings.html\" target=\"_blank\">http:\/\/webont.org\/owled\/2007\/Proceedings.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Igniting+the+OWL+1.1+Touch+Paper%3A+The+OWL+API&rft.jtitle=OWL%3A+Experiences+and+Directions+2007&rft.aulast=Horridge%2C+M.%3B+Bechhofer%2C+S.%3B+Noppens%2C+O.&rft.au=Horridge%2C+M.%3B+Bechhofer%2C+S.%3B+Noppens%2C+O.&rft.date=2007&rft.pages=1%E2%80%939&rft_id=http%3A%2F%2Fwebont.org%2Fowled%2F2007%2FProceedings.html&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubSlimmer-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GitHubSlimmer_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/enanomapper\/slimmer\/\" target=\"_blank\">\"enanomapper \/ slimmer\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/enanomapper\/slimmer\/\" target=\"_blank\">https:\/\/github.com\/enanomapper\/slimmer\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 May 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=enanomapper+%2F+slimmer&rft.atitle=GitHub&rft_id=https%3A%2F%2Fgithub.com%2Fenanomapper%2Fslimmer%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HastingsEnano15-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HastingsEnano15_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hastings, J.; Jeliazkova, N.; Owen, G. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4374589\" target=\"_blank\">\"eNanoMapper: harnessing ontologies to enable data integration for nanomaterial risk assessment\"<\/a>. <i>Journal of Biomedical Semantics<\/i> <b>6<\/b>: 10. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13326-015-0005-5\" target=\"_blank\">10.1186\/s13326-015-0005-5<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4374589\/\" target=\"_blank\">PMC4374589<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25815161\" target=\"_blank\">25815161<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4374589\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4374589<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=eNanoMapper%3A+harnessing+ontologies+to+enable+data+integration+for+nanomaterial+risk+assessment&rft.jtitle=Journal+of+Biomedical+Semantics&rft.aulast=Hastings%2C+J.%3B+Jeliazkova%2C+N.%3B+Owen%2C+G.+et+al.&rft.au=Hastings%2C+J.%3B+Jeliazkova%2C+N.%3B+Owen%2C+G.+et+al.&rft.date=2015&rft.volume=6&rft.pages=10&rft_id=info:doi\/10.1186%2Fs13326-015-0005-5&rft_id=info:pmc\/PMC4374589&rft_id=info:pmid\/25815161&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4374589&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StuckyOnto17-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StuckyOnto17_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stucky, B.J.; Luc, A. (2017). \"OntoPilot: New Software to Simplify and Accelerate Ontology Development and Deployment\". <i>Biodiversity Information Science and Standards<\/i> <b>1<\/b>: e20192. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Ftdwgproceedings.1.20192\" target=\"_blank\">10.3897\/tdwgproceedings.1.20192<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=OntoPilot%3A+New+Software+to+Simplify+and+Accelerate+Ontology+Development+and+Deployment&rft.jtitle=Biodiversity+Information+Science+and+Standards&rft.aulast=Stucky%2C+B.J.%3B+Luc%2C+A.&rft.au=Stucky%2C+B.J.%3B+Luc%2C+A.&rft.date=2017&rft.volume=1&rft.pages=e20192&rft_id=info:doi\/10.3897%2Ftdwgproceedings.1.20192&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubOwlTools-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GitHubOwlTools_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-GitHubOwlTools_14-1\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/owlcollab\/owltools\" target=\"_blank\">\"owlcollab \/ owltools\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/owlcollab\/owltools\" target=\"_blank\">https:\/\/github.com\/owlcollab\/owltools<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 May 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=owlcollab+%2F+owltools&rft.atitle=GitHub&rft_id=https%3A%2F%2Fgithub.com%2Fowlcollab%2Fowltools&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubOwlTools-Oort-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GitHubOwlTools-Oort_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-GitHubOwlTools-Oort_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/owlcollab\/owltools\/wiki\/Oort-Intro\" target=\"_blank\">\"owlcollab \/ owltools : Oort Intro\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/owlcollab\/owltools\/wiki\/Oort-Intro\" target=\"_blank\">https:\/\/github.com\/owlcollab\/owltools\/wiki\/Oort-Intro<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 May 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=owlcollab+%2F+owltools+%3A+Oort+Intro&rft.atitle=GitHub&rft_id=https%3A%2F%2Fgithub.com%2Fowlcollab%2Fowltools%2Fwiki%2FOort-Intro&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubROBOT-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GitHubROBOT_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-GitHubROBOT_16-1\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/ontodev\/robot\" target=\"_blank\">\"ontodev \/ robot\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/ontodev\/robot\" target=\"_blank\">https:\/\/github.com\/ontodev\/robot<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 October 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ontodev+%2F+robot&rft.atitle=GitHub&rft_id=https%3A%2F%2Fgithub.com%2Fontodev%2Frobot&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OvertonROBOT15-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OvertonROBOT15_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Overton, J.A.; Dietze, H.; Essaid, S. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ceur-ws.org\/Vol-1515\/\" target=\"_blank\">\"ROBOT: A command-line tool for ontology development\"<\/a>. <i>Proceedings of the 2015 International Conference on Biomedical Ontology<\/i>: 1\u20132<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ceur-ws.org\/Vol-1515\/\" target=\"_blank\">http:\/\/ceur-ws.org\/Vol-1515\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ROBOT%3A+A+command-line+tool+for+ontology+development&rft.jtitle=Proceedings+of+the+2015+International+Conference+on+Biomedical+Ontology&rft.aulast=Overton%2C+J.A.%3B+Dietze%2C+H.%3B+Essaid%2C+S.+et+al.&rft.au=Overton%2C+J.A.%3B+Dietze%2C+H.%3B+Essaid%2C+S.+et+al.&rft.date=2015&rft.pages=1%E2%80%932&rft_id=http%3A%2F%2Fceur-ws.org%2FVol-1515%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarrollJena04-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CarrollJena04_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-CarrollJena04_18-1\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Carroll, J.J.; Dickinson. I.; Dollin, C. et al. (2004). \"Jena: Implementing the semantic web recommendations\". <i>WWW Alt. '04: Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters<\/i>: 74\u201383. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1013367.1013381\" target=\"_blank\">10.1145\/1013367.1013381<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Jena%3A+Implementing+the+semantic+web+recommendations&rft.jtitle=WWW+Alt.+%2704%3A+Proceedings+of+the+13th+international+World+Wide+Web+conference+on+Alternate+track+papers+%26+posters&rft.aulast=Carroll%2C+J.J.%3B+Dickinson.+I.%3B+Dollin%2C+C.+et+al.&rft.au=Carroll%2C+J.J.%3B+Dickinson.+I.%3B+Dollin%2C+C.+et+al.&rft.date=2004&rft.pages=74%E2%80%9383&rft_id=info:doi\/10.1145%2F1013367.1013381&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ApacheCommons-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ApacheCommons_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/commons.apache.org\/proper\/commons-cli\/\" target=\"_blank\">\"Commons CLI\"<\/a>. <i>Apache Commons<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/commons.apache.org\/proper\/commons-cli\/\" target=\"_blank\">https:\/\/commons.apache.org\/proper\/commons-cli\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 May 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Commons+CLI&rft.atitle=Apache+Commons&rft_id=https%3A%2F%2Fcommons.apache.org%2Fproper%2Fcommons-cli%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JUnit-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JUnit_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/junit.org\/junit4\/\" target=\"_blank\">\"JUnit\"<\/a>. <i>junit.org<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/junit.org\/junit4\/\" target=\"_blank\">https:\/\/junit.org\/junit4\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 May 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=JUnit&rft.atitle=junit.org&rft_id=https%3A%2F%2Fjunit.org%2Fjunit4%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TravisCI-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TravisCI_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-TravisCI_21-1\">21.1<\/a><\/sup> <sup><a href=\"#cite_ref-TravisCI_21-2\">21.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/travis-ci.org\/\" target=\"_blank\">\"Travis CI\"<\/a>. Travis CI, GmbH<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/travis-ci.org\/\" target=\"_blank\">https:\/\/travis-ci.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 May 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Travis+CI&rft.atitle=&rft.pub=Travis+CI%2C+GmbH&rft_id=https%3A%2F%2Ftravis-ci.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KazakovTheIncr13-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KazakovTheIncr13_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kazakov, Y.; Kr\u00f6tzsch, M.; Siman\u010d\u00edk, F. (2014). \"The Incredible ELK\". <i>Journal of Automated Reasoning<\/i> <b>53<\/b>: 1\u201361. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10817-013-9296-3\" target=\"_blank\">10.1007\/s10817-013-9296-3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Incredible+ELK&rft.jtitle=Journal+of+Automated+Reasoning&rft.aulast=Kazakov%2C+Y.%3B+Kr%C3%B6tzsch%2C+M.%3B+Siman%C4%8D%C3%ADk%2C+F.&rft.au=Kazakov%2C+Y.%3B+Kr%C3%B6tzsch%2C+M.%3B+Siman%C4%8D%C3%ADk%2C+F.&rft.date=2014&rft.volume=53&rft.pages=1%E2%80%9361&rft_id=info:doi\/10.1007%2Fs10817-013-9296-3&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShearerHermit08-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShearerHermit08_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shearer, R.; Motik, B.; Horrocks, I. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ceur-ws.org\/Vol-432\/\" target=\"_blank\">\"HermiT: A Highly-Efficient OWL Reasoner\"<\/a>. <i>OWL: Experiences and Directions 2008<\/i>: 1\u201310<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ceur-ws.org\/Vol-432\/\" target=\"_blank\">http:\/\/ceur-ws.org\/Vol-432\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=HermiT%3A+A+Highly-Efficient+OWL+Reasoner&rft.jtitle=OWL%3A+Experiences+and+Directions+2008&rft.aulast=Shearer%2C+R.%3B+Motik%2C+B.%3B+Horrocks%2C+I.&rft.au=Shearer%2C+R.%3B+Motik%2C+B.%3B+Horrocks%2C+I.&rft.date=2008&rft.pages=1%E2%80%9310&rft_id=http%3A%2F%2Fceur-ws.org%2FVol-432%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubEMR-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GitHubEMR_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/owlcollab\/expression-materializing-reasoner\" target=\"_blank\">\"owlcollab \/ expression-materializing-reasoner\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/owlcollab\/expression-materializing-reasoner\" target=\"_blank\">https:\/\/github.com\/owlcollab\/expression-materializing-reasoner<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 October 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=owlcollab+%2F+expression-materializing-reasoner&rft.atitle=GitHub&rft_id=https%3A%2F%2Fgithub.com%2Fowlcollab%2Fexpression-materializing-reasoner&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HastingsTheChe13-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HastingsTheChe13_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hastings, J.; de Matos, P.; Dekker, A. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531142\" target=\"_blank\">\"The ChEBI reference database and ontology for biologically relevant chemistry: Enhancements for 2013\"<\/a>. <i>Nucleic Acids Research<\/i> <b>41<\/b> (DB1): D456-63. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgks1146\" target=\"_blank\">10.1093\/nar\/gks1146<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3531142\/\" target=\"_blank\">PMC3531142<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23180789\" target=\"_blank\">23180789<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531142\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531142<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+ChEBI+reference+database+and+ontology+for+biologically+relevant+chemistry%3A+Enhancements+for+2013&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Hastings%2C+J.%3B+de+Matos%2C+P.%3B+Dekker%2C+A.+et+al.&rft.au=Hastings%2C+J.%3B+de+Matos%2C+P.%3B+Dekker%2C+A.+et+al.&rft.date=2013&rft.volume=41&rft.issue=DB1&rft.pages=D456-63&rft_id=info:doi\/10.1093%2Fnar%2Fgks1146&rft_id=info:pmc\/PMC3531142&rft_id=info:pmid\/23180789&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3531142&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HillDove13-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HillDove13_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hill, D.P.; Adams, N.; Bada, M.A. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3733925\" target=\"_blank\">\"Dovetailing biology and chemistry: Integrating the Gene Ontology with the ChEBI chemical ontology\"<\/a>. <i>BMC Genomics<\/i> <b>14<\/b>: 513. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2164-14-513\" target=\"_blank\">10.1186\/1471-2164-14-513<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3733925\/\" target=\"_blank\">PMC3733925<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23895341\" target=\"_blank\">23895341<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3733925\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3733925<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Dovetailing+biology+and+chemistry%3A+Integrating+the+Gene+Ontology+with+the+ChEBI+chemical+ontology&rft.jtitle=BMC+Genomics&rft.aulast=Hill%2C+D.P.%3B+Adams%2C+N.%3B+Bada%2C+M.A.+et+al.&rft.au=Hill%2C+D.P.%3B+Adams%2C+N.%3B+Bada%2C+M.A.+et+al.&rft.date=2013&rft.volume=14&rft.pages=513&rft_id=info:doi\/10.1186%2F1471-2164-14-513&rft_id=info:pmc\/PMC3733925&rft_id=info:pmid\/23895341&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3733925&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GrauModular08-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GrauModular08_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Grau, B.C.; Horrocks, I.; Kazakov, Y. et al. (2008). \"Modular Reuse of Ontologies: Theory and Practice\". <i>Journal of Artificial Intelligence Research<\/i> <b>31<\/b>: 273\u2013318. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1613%2Fjair.2375\" target=\"_blank\">10.1613\/jair.2375<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Modular+Reuse+of+Ontologies%3A+Theory+and+Practice&rft.jtitle=Journal+of+Artificial+Intelligence+Research&rft.aulast=Grau%2C+B.C.%3B+Horrocks%2C+I.%3B+Kazakov%2C+Y.+et+al.&rft.au=Grau%2C+B.C.%3B+Horrocks%2C+I.%3B+Kazakov%2C+Y.+et+al.&rft.date=2008&rft.volume=31&rft.pages=273%E2%80%93318&rft_id=info:doi\/10.1613%2Fjair.2375&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarrisSPARQL13-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HarrisSPARQL13_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Harris, S.; Seaborne, A.; Prud'hommeaux, E., ed. (21 March 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.w3.org\/TR\/sparql11-query\/\" target=\"_blank\">\"SPARQL 1.1 Query Language\"<\/a>. <i>w3.org<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.w3.org\/TR\/sparql11-query\/\" target=\"_blank\">https:\/\/www.w3.org\/TR\/sparql11-query\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 May 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SPARQL+1.1+Query+Language&rft.atitle=w3.org&rft.date=21+March+2013&rft_id=https%3A%2F%2Fwww.w3.org%2FTR%2Fsparql11-query%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MungallTaking11-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MungallTaking11_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mungall, C.; Ruttenberg, A.; Osumi-Sutherland, D. (2011). \"Taking shortcuts with OWL using safe macros\". <i>Nature Precedings<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnpre.2011.5292.2\" target=\"_blank\">10.1038\/npre.2011.5292.2<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Taking+shortcuts+with+OWL+using+safe+macros&rft.jtitle=Nature+Precedings&rft.aulast=Mungall%2C+C.%3B+Ruttenberg%2C+A.%3B+Osumi-Sutherland%2C+D.&rft.au=Mungall%2C+C.%3B+Ruttenberg%2C+A.%3B+Osumi-Sutherland%2C+D.&rft.date=2011&rft_id=info:doi\/10.1038%2Fnpre.2011.5292.2&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Osumi-SutherlandDead17-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Osumi-SutherlandDead17_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Osumi-Sutherland, D.; Courtot, M.; Balhoff, J.P. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5460348\" target=\"_blank\">\"Dead simple OWL design pattern\"<\/a>. <i>Journal of Biomedical Semantics<\/i> <b>8<\/b> (1): 18. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13326-017-0126-0\" target=\"_blank\">10.1186\/s13326-017-0126-0<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5460348\/\" target=\"_blank\">PMC5460348<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28583177\" target=\"_blank\">28583177<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5460348\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5460348<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Dead+simple+OWL+design+pattern&rft.jtitle=Journal+of+Biomedical+Semantics&rft.aulast=Osumi-Sutherland%2C+D.%3B+Courtot%2C+M.%3B+Balhoff%2C+J.P.+et+al.&rft.au=Osumi-Sutherland%2C+D.%3B+Courtot%2C+M.%3B+Balhoff%2C+J.P.+et+al.&rft.date=2017&rft.volume=8&rft.issue=1&rft.pages=18&rft_id=info:doi\/10.1186%2Fs13326-017-0126-0&rft_id=info:pmc\/PMC5460348&rft_id=info:pmid\/28583177&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5460348&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GNUMake-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GNUMake_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.gnu.org\/software\/make\/\" target=\"_blank\">\"GNU Make\"<\/a>. <i>GNU - Software<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.gnu.org\/software\/make\/\" target=\"_blank\">https:\/\/www.gnu.org\/software\/make\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 May 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GNU+Make&rft.atitle=GNU+-+Software&rft_id=https%3A%2F%2Fwww.gnu.org%2Fsoftware%2Fmake%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-C.C3.B4t.C3.A9TheOnt10-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-C.C3.B4t.C3.A9TheOnt10_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">C\u00f4t\u00e9, R.; Reisinger, F.; Martens, L. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2896109\" target=\"_blank\">\"The Ontology Lookup Service: Bigger and better\"<\/a>. <i>Nucleic Acids Research<\/i> <b>38<\/b> (WSI): W155-60. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkq331\" target=\"_blank\">10.1093\/nar\/gkq331<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2896109\/\" target=\"_blank\">PMC2896109<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20460452\" target=\"_blank\">20460452<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2896109\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2896109<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Ontology+Lookup+Service%3A+Bigger+and+better&rft.jtitle=Nucleic+Acids+Research&rft.aulast=C%C3%B4t%C3%A9%2C+R.%3B+Reisinger%2C+F.%3B+Martens%2C+L.+et+al.&rft.au=C%C3%B4t%C3%A9%2C+R.%3B+Reisinger%2C+F.%3B+Martens%2C+L.+et+al.&rft.date=2010&rft.volume=38&rft.issue=WSI&rft.pages=W155-60&rft_id=info:doi\/10.1093%2Fnar%2Fgkq331&rft_id=info:pmc\/PMC2896109&rft_id=info:pmid\/20460452&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2896109&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubODK-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GitHubODK_33-0\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-GitHubODK_33-1\">33.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/INCATools\/ontology-development-kit\/\" target=\"_blank\">\"INCATools \/ ontology-development-kit\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/INCATools\/ontology-development-kit\/\" target=\"_blank\">https:\/\/github.com\/INCATools\/ontology-development-kit\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 October 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=INCATools+%2F+ontology-development-kit&rft.atitle=GitHub&rft_id=https%3A%2F%2Fgithub.com%2FINCATools%2Fontology-development-kit%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DHObolibrary-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DHObolibrary_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/hub.docker.com\/u\/obolibrary\" target=\"_blank\">\"obolibrary\"<\/a>. <i>DockerHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/hub.docker.com\/u\/obolibrary\" target=\"_blank\">https:\/\/hub.docker.com\/u\/obolibrary<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 October 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=obolibrary&rft.atitle=DockerHub&rft_id=https%3A%2F%2Fhub.docker.com%2Fu%2Fobolibrary&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BandrowskiTheOnt16-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BandrowskiTheOnt16_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bandrowski, A.; Brinkman, R.; Brochhausen, M. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4851331\" target=\"_blank\">\"The Ontology for Biomedical Investigations\"<\/a>. <i>PLoS One<\/i> <b>11<\/b> (4): e0154556. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0154556\" target=\"_blank\">10.1371\/journal.pone.0154556<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4851331\/\" target=\"_blank\">PMC4851331<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27128319\" target=\"_blank\">27128319<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4851331\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4851331<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Ontology+for+Biomedical+Investigations&rft.jtitle=PLoS+One&rft.aulast=Bandrowski%2C+A.%3B+Brinkman%2C+R.%3B+Brochhausen%2C+M.+et+al.&rft.au=Bandrowski%2C+A.%3B+Brinkman%2C+R.%3B+Brochhausen%2C+M.+et+al.&rft.date=2016&rft.volume=11&rft.issue=4&rft.pages=e0154556&rft_id=info:doi\/10.1371%2Fjournal.pone.0154556&rft_id=info:pmc\/PMC4851331&rft_id=info:pmid\/27128319&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4851331&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BelloDisease18-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BelloDisease18_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bello, S.M.; Shimoyama, M.; Mitraka, E. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5897730\" target=\"_blank\">\"Disease Ontology: improving and unifying disease annotations across species\"<\/a>. <i>Disease Models and Mechanisms<\/i> <b>11<\/b> (3): dmm032839. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1242%2Fdmm.032839\" target=\"_blank\">10.1242\/dmm.032839<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5897730\/\" target=\"_blank\">PMC5897730<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29590633\" target=\"_blank\">29590633<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5897730\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5897730<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Disease+Ontology%3A+improving+and+unifying+disease+annotations+across+species&rft.jtitle=Disease+Models+and+Mechanisms&rft.aulast=Bello%2C+S.M.%3B+Shimoyama%2C+M.%3B+Mitraka%2C+E.+et+al.&rft.au=Bello%2C+S.M.%3B+Shimoyama%2C+M.%3B+Mitraka%2C+E.+et+al.&rft.date=2018&rft.volume=11&rft.issue=3&rft.pages=dmm032839&rft_id=info:doi\/10.1242%2Fdmm.032839&rft_id=info:pmc\/PMC5897730&rft_id=info:pmid\/29590633&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5897730&rfr_id=info:sid\/en.wikipedia.org:Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference. A few citations were turned into simple inline links to content, where appropriate.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204345\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.783 seconds\nReal time usage: 1.078 seconds\nPreprocessor visited node count: 27082\/1000000\nPreprocessor generated node count: 34954\/1000000\nPost\u2010expand include size: 192962\/2097152 bytes\nTemplate argument size: 64802\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 740.066 1 - -total\n 88.12% 652.145 1 - Template:Reflist\n 77.32% 572.209 36 - Template:Citation\/core\n 58.88% 435.777 24 - Template:Cite_journal\n 21.93% 162.319 12 - Template:Cite_web\n 6.68% 49.443 1 - Template:Infobox_journal_article\n 6.65% 49.226 39 - Template:Citation\/identifier\n 6.40% 47.337 1 - Template:Infobox\n 5.33% 39.455 47 - Template:Citation\/make_link\n 4.30% 31.853 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11418-0!*!0!!en!5!* and timestamp 20200707204344 and revision id 37970\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows\">https:\/\/www.limswiki.org\/index.php\/Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","2351ba0d1aa807936bc9922c2403b7f2_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/83\/Fig1_Jackson_BMCBioinformatics2019_20.png"],"2351ba0d1aa807936bc9922c2403b7f2_timestamp":1594154624,"45b5e684bad1c44cd9a1cb43bb6b311e_type":"article","45b5e684bad1c44cd9a1cb43bb6b311e_title":"One tool to find them all: A case of data integration and querying in a distributed LIMS platform (Grand et al. 2019)","45b5e684bad1c44cd9a1cb43bb6b311e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform","45b5e684bad1c44cd9a1cb43bb6b311e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:One tool to find them all: A case of data integration and querying in a distributed LIMS platform\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nOne tool to find them all: A case of data integration and querying in a distributed LIMS platformJournal\n \nDatabaseAuthor(s)\n \nGrand, Alberto; Geda, Emanuele; Mignone, Andrea; Bertotti, Andrea; Fiori, AlessandroAuthor affiliation(s)\n \nCandiolo Cancer Institute, University of TorinoPrimary contact\n \nEmail: alessandro dot fiori at ircc dot itYear published\n \n2019Volume and issue\n \n219Page(s)\n \nbaz004DOI\n \n10.1093\/database\/baz004ISSN\n \n1758-0463Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/academic.oup.com\/database\/article\/doi\/10.1093\/database\/baz004\/5304001Download\n \nhttps:\/\/academic.oup.com\/database\/article-pdf\/doi\/10.1093\/database\/baz004\/27643896\/baz004.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 LAS \n\n3.1 Data models \n3.2 Functionalities \n\n\n4 MDDM \n\n4.1 Architecture \n4.2 Logical design \n4.3 Query generator \n4.4 Query execution \n4.5 Visualization tools \n\n\n5 Platform usage \n6 Conclusion \n7 Acknowledgements \n\n7.1 System availability \n7.2 Conflict of interest \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nIn recent years, laboratory information management systems (LIMS) have been growing from mere inventory systems into increasingly comprehensive software platforms, spanning functionalities as diverse as data search, annotation, and analysis. In 2011, our institution started a LIMS project named the Laboratory Assistant Suite with the purpose of assisting researchers throughout all of their laboratory activities, providing graphical tools to support decision-making tasks and building complex analyses on integrated data. The modular architecture of the system exploits multiple databases with different technologies. To provide an efficient and easy tool for retrieving information of interest, we developed the Multi-Dimensional Data Manager (MDDM). By means of intuitive interfaces, scientists can execute complex queries without any knowledge of query languages or database structures, and easily integrate heterogeneous data stored in multiple databases. Together with the other software modules making up the platform, the MDDM has helped improve the overall quality of the data, substantially reduced the time spent with manual data entry and retrieval, and ultimately broadened the spectrum of interconnections among the data, offering novel perspectives to biomedical analysts.\n\nIntroduction \nThe introduction of automation and high-throughput technologies in laboratory environments has raised diverse issues related to the amount and heterogeneity of the data produced, the adoption of robust procedures for sample tracking, and the management of computer-based workflows needed to process and analyze the raw data. Laboratory information management systems (LIMS) have gained increasing popularity because they can ensure good levels of quality control over laboratory activities and efficiently handle the large amounts of data produced.[1]\nLIMS aim at assisting the researchers in their daily laboratory practice, improving the accessibility of instruments, and tracking biological samples and their related information.\nIn the past decade, several open-source as well as proprietary LIMS have been developed. Commercial solutions are typically large, complex, and feature-rich products designed to easily support large laboratories. Their license fees can be prohibitive, and extra features may come at additional costs.[2] To reduce these costs, the last generation of commercial LIMS adopt web-oriented software technologies, particularly the software-as-a-service distribution model, which reduces the customer\u2019s final expenditure on license fees, hardware, and maintenance. Examples of commercial solutions include STARLIMS[3], Exemplar LIMS[4], and LabVantage.[5]\nCommercial LIMS tend to offer features based on common laboratory procedures and best practices, which may not fit highly specific settings well. For instance, LabVantage provides a large set of features, such as sample and batch management, quality control, advanced storage and logistics, and task scheduling. However, the life cycle of xenopatients (i.e., biological models for cancer research based on the transplantation of human tumors in mice) is not available in the standard software and should be implemented as a custom module by the software developer. Another issue that affects commercial LIMSs is the management and standardization of genomic data. To the best of our knowledge, these systems do not exploit any knowledge base related to the genomic data and do not provide any validation and analysis of different genomic data stored in the system.\nOther open-source solutions like Galaxy[6][7] focus instead on specific sub-domains, addressing DNA sequencing and annotation, or SeqWare[8], which tracks in vivo and in vitro experiments and allows for complex analysis workflows.\nFor this reason, many institutions have invested in the development of in-house solutions and\/or have adapted open-source projects to their own requirements. In this way, the developed solutions can provide functionality that meet the specific needs of the researchers in their institution laboratories. From an engineering perspective, developing in-house solutions may also permit the exploration and adoption of new technologies, in order to define better data models and improve system performance.\nTo address a substantial mismatch between the LIMS solutions on offer and the functional requirements dictated by research practice, the Institute for Cancer Research at Candiolo (Italy) started to implement its own LIMS, named the Laboratory Assistant Suite (LAS) platform, in 2011.[9] The main purpose of the platform was to assist researchers in different laboratory and research activities, allowing management of different kinds of raw data (e.g., biological, molecular), tracking experimental data, supporting decision-making tasks, and integrating heterogeneous data for complex analyses. As development progressed, several new features and modules were included to (i) track clinical data, (ii) include support to the newest technologies exploited for molecular experiments, and (iii) standardize the description of genomic data by means of semantic web technologies. Thanks to these new features, scientists can gain better insight into tumor development by jointly studying the clinical evolution of the disease and the experimental results derived from in vivo and in vitro experimentation. The experimental pipelines exploited in the translational research context are the primary focus of the LAS, which targets the standardization of the genomic data to allow a comparison of results coming from different technologies.\nStill, unlike the other commercial and open-source platforms, the LAS makes an attempt at covering a wide range of diverse laboratory procedures and, thanks to its versatile and general-purpose structure, it can be extended to support new ones with limited effort.\nThanks to the vast variety of different experimental technologies supported by the LAS and their high level of specificity, large amounts of heterogeneous and complex information are collected in separate databases. To enable the users to extract and correlate information from the different databases exploited by the platform, a Multi-Dimensional Data Manager (MDMM) module was developed. The module takes care of merging data from the different LAS databases and provides a simple graphical user interface to extract the information of interest without any knowledge of a query language. A tool to visualize biological entities and their related information with a hierarchical tree structure is also available, while other powerful visualization tools are currently under development. To the best of our knowledge, no similar tools applied to biological data and distributed databases exist.\nThis paper presents the main characteristics of the LAS and its exploitation in the research laboratories of the Institute for Cancer Research at Candiolo by its researchers and research partners. Next, the main features of the MDDM are described. Afterwards, current and future research directions are presented.\n\nLAS \nTranslational research aims at enhancing patient care and transferring scientific discoveries from the laboratory to a real clinical context. It is a kind of metaphorical scientific cycle from bench to bedside and back again through complex iterative processes, operating between laboratory (i.e., preclinical research) and clinic. To the aim of managing and integrating preclinical and clinical information, a robust but flexible data management platform is needed. In particular, different types of information (e.g., biological data, molecular data, procedure tracking data, and sample tracking data)\u2014some of which can be highly complex\u2014should be independently managed by the platform but, at the same time, interconnected to permit integrated analyses.\nThe LAS platform is freely available upon request to the authors. The software is distributed by means of a Docker-based approach to allow interested organizations to configure it according to their constraints. Moreover, the use of Docker allows system administrators to run the software on different servers using the Docker Swarm configuration for balancing the workload, as well as the associated data resources. We usually recommend installing the LAS on at least two servers, one dedicated to the containers running the software and the other for databases. The servers characteristics depend on several aspects, such as the number of simultaneously logged users, the number of biological entities tracked, and the dimension of raw data stored. We suggest as initial setup a server with at least 16 GB of RAM and a storage space of 2 TB. Interested users may refer to the video tutorials (available at http:\/\/lasrep.ircc.it\/) to explore main system features and as a reference guide during its usage. The documentation of the platform is provided with the software and can be downloaded from the documentation section of the LAS instance of the Institute for Cancer Research at Candiolo.\nWhat follows is a description of the data architecture and the main functionalities included in the platform.\n\nData models \nThe LAS platform has been developed using different database technologies to fit the needs of the application, and to handle in a suitable way the heterogeneous data characteristics.\nThe platform makes use of relational databases to track biological entities and their properties, as well as the information about the various experimental procedures. Since the platform includes different modules managing substantially different types of entities and\/or specific laboratory procedures, different database instances are exploited. The core biological entities (i.e., Aliquot, Biomouse, and Cell Line) are identified by a unique and mnemonic key named GenealogyID that encodes relevant information regarding the history of the entity. This key is automatically generated by the LAS platform through formal rules and may be used to link the data across the databases.\nParallel to the relational databases storing operating data, a graph database is exploited. It is used to represent the complex inherent hierarchy of biological entities and their relationships. Being able to easily and efficiently reconstruct the genealogical tree of each entity is indeed an essential feature of the platform, allowing the user to perform ad hoc queries and to isolate specific sub-trees of biological entities involved in the experimental pipeline. Moreover, the graph database has been exploited to store a knowledge base for the heterogeneous domains managed by the LAS modules. By using a graph representation, all these domains can be easily interconnected, while the knowledge base can be continuously updated and augmented with new layers of information and different levels of abstraction (e.g., proteomics, clinical, etc.). Finally, a social network of users and research groups using the LAS platform is also stored in the graph, to model data ownership, resolve data access conflicts, and manage data sharing and collaboration among different groups or users.\nA document database, MongoDB, is also used to store files associated with biological entities and metadata generated by both the LAS Genomic Data Manager modules and the MDDM. The latter usage will be discussed in detail in the next section.\n\nFunctionalities \nThe LAS architecture includes a set of software modules, i.e., fully-fledged web applications, each addressing a different type of biological entity and its associated experimental procedures. Modules may interact with each other by means of web application programming interfaces (APIs), e.g., to exchange data and\/or to carry out operations that span multiple entities or domains. (The modules currently included in the LAS platform are described in the following.)\nEven if the platform has been developed since 2011, we always took into account security issues during the design and development processes of the platform. In this way, our software is compliant with the constraints of the General Data Protection Regulation (GDPR), with enforcement beginning on May 25, 2018. Indeed, the management of data produced by different users and\/or groups requires that the access to functionalities and information are restricted according to several criteria such as group and\/or project membership, and user role. For these reasons, the platform manages users and their privileges following these concepts:\n\n Working Group: A Working Group (WG) is a set of users in the LAS platform that work together toward a specific goal (e.g., project, research activity). The data produced by the users of the same group are private, unless they intentionally share data with other groups.\n User Profiles: Each user belonging to a WG has a set of permissions to access the LAS functionalities they have been enabled to use. These functionalities are defined according to the role selected during the user registration process. The manager of their WG or the system administrator can assign new functionalities upon request.\nTo collect data, the user is required to specify the informed consent signed by the patient for specific research activities (e.g., preclinical trials) involving personal samples and information. This document is defined by a committee to accomplish all the constraints included in the GDPR. Since the data are collected for research purposes, the patient can only revoke the usage of the biological samples, but not the information (e.g., experimental results) collected by the researchers. Only the researchers that are included in the research project can manage these samples and track the experimental processes according to their profile. The platform tracks all the procedures performed by each user in order to identify malicious usage of the software.\nFor each patient, the Clinical module tracks both contextual information (i.e., personal data, Medical Center of the Trial, etc.) and relevant clinical events through a case report form. All data are related to the relative informed consent that grants data and specimen sampling.\nThe BioBanking Management module covers a wide range of activities, including management of biological samples and associated pathological information, as well as support for a number of laboratory-related procedures. For instance, the module can handle the collection of biological material from surgical intervention and the acquisition of aliquots from external laboratories. Aliquots stored in the system are characterized by features such as tumor type (e.g., colorectal), tissue type (e.g., liver metastasis), source hospital, or laboratory and pathological information. Measurement of aliquot physical characteristics such as volume, concentration, purity, and quality can be tracked by the module, as well as the derivation of new biological materials (e.g., DNA and cDNA) and the planning of molecular experiments.\nThe biological material used in our laboratories is stored by means of several types of containers (e.g., freezers, racks, plates, and tubes). Their mutual interactions (i.e., which types of containers can host other containers) can change according to characteristics such as the layout and the laboratory procedure. Additionally, the Storage Management module allows managing any kind of container by defining and applying different rules to them, and it tracks the relationships between the containers and the biological entities.\nDifferent types of molecular analyses can be conducted on biological samples, to investigate various aspects of their genetic constituents that may have an impact on the development of oncogenic behavior. For instance, biologists may be interested in analyzing mutations for a target gene involved in tumor proliferation. In an effort to closely track the translational research pipeline from the collection of samples to their analysis, the LAS provides support to tracking the most frequently used molecular profiling techniques in our institution {e.g., Sanger sequencing, real-time polymerase chain reaction (PCR), and Sequenom). Each molecular module queries the knowledge base of the Genomic Annotation Manager (GAM) to retrieve the description of its reagents, as well as a specification of all possible alterations (e.g., sequence alterations and gene copy number variations) known in the literature, to allow both the experiment definition and the evaluation of experimental results.\nThe GAM provides a higher-level, qualitative insight into the genomic features of biological samples. This information is shaped in the form of annotations, i.e., a set of semantic labels attached to a sample, pointing out some of its relevant features. To ensure semantic coherence and adopt a standardized nomenclature, all relevant concepts from the genomic and biological domains used for labeling samples have been drawn from a number of public, freely accessible databases and ontologies.[10][11][12][13] This information has been structured into a knowledge base, modeled as a graph, and stored in a graph database.[14] Concepts are interlinked with one another according to both general-purpose semantic relationships such as containment (\"part of\") or generalization (\"is a\"), and domain-specific relationships (e.g., indicating an underlying biochemical process, as in \"is transcribed from\"). New concepts and relationships, as well as new domains of interest, may be added or layered as needed to account for novel findings and broaden the spectrum of investigation. Within the GAM, every annotation is a semantic statement establishing a relationship, expressed by means of a predicate, between a biological sample (the subject of the statement) and a concept (the object of the statement), such as a genetic mutation. It is represented within the graph database as a node of type \"annotation\" with a pair of incoming and outgoing edges, one linking the biological sample to the annotation node by means of a has_annotation relationship, and the other linking the annotation node to the reference node in the knowledge base by means of a has_reference relationship. The annotation node is often linked to other nodes, such as the process that produced the annotation or the raw experimental data.\nFinally, the platform allows tracking in vivo and in vitro experiments. Studies that are \"in vivo\" (Latin for \"within the living\") are those in which the effects of various biological entities are tested on whole, living organisms, usually animals (e.g., xenopatients). This kind of experiment is normally exploited to test drug therapies and expand the collection of biological samples. We based our development on the model described by Bertotti et al.[15] to manage immunocompromised animals and monitor the xenopatient life cycle, from their acquisition by the research institute to their death. Unlike in vivo experiments, in vitro studies are performed with cells or biological molecules studied outside their normal biological context. The LAS platform defines a cell line as the set of biological entities that are generated from the same biological entity and are under the same experimental conditions. The experimental conditions are defined by the protocols that describe the type of process (i.e., adherent, suspend, and organoid) and the set of culturing conditions applied (e.g., nutrients and chemicals, hormones\/growth factors, and antibiotics). In addition, the platform allows the management of the generation and thawing procedures of cell lines.\n\nMDDM \nBesides tracking experimental procedures and recording all the data related to biological entities, the retrieval of the information tracked by the platform is fundamental to discover new knowledge related to tumors. Due to the large number of laboratory activities and biological entities managed by the LAS with a high level of specificity, and to streamline the development and maintenance process over time, each LAS module operates on a separate database instance, storing both operational data and domain-specific knowledge, and partially replicating key information from other LAS modules. As a result, while each instance focuses on its own specific subset of functionalities, no one database can provide a comprehensive insight into the data. Nonetheless, building actionable knowledge requires the integration of heterogeneous information to establish connections among different and related biological entities, or to explore different facets of the same entities, both for operational and for research purposes. For instance, identifying the number of tubes stored in each plate of a given freezer and their content may be required for managing physical storage equipment. From a researcher\u2019s perspective, the study of the evolution of the tumor mass in a mouse under pharmacological treatment (monitored by means of in vivo experimentation and tracked by the Xenograft Management module) must be traced back to the originating patient (managed by the Clinical module), and should be further correlated to the mouse genomic mutational status and gene expression levels (collected by the set of Molecular modules and annotated by the Genomic Annotation module). The MDDM addresses the issue of extracting all information of interest stored in the databases of each LAS module, providing the end user with an intuitive graphical tool for building customized queries, with a unified view on the entire collection of databases. In addition, the MDDM can be exploited to run programmatic queries via a set of web APIs, enabling other LAS modules, as well as other external applications, to retrieve data of interest.\n\nArchitecture \nThe MDDM has a distributed architecture. Its main components, shown in Figure 1, are the central MDDM Data Integrator (DI) module and the distributed MDDM Data Provider (DP) modules. The DI module provides a unified logical view on the databases of the other modules, thus acting as a (read-only) federated database.[16] The DP modules rely on a standalone application layer that must be installed within each LAS module, allowing the LAS modules to join the federated database. Interactions between the DI and the DP modules occur via two distinct sets of APIs.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. MDMM architecture\n\n\n\nThe MDDM DP APIs are called by the central DI module. They provide an interface to perform the following tasks:\n1. Retrieve schema information from each LAS module\u2019s database. This operation is executed once at setup time, when the LAS module is initially registered as a member of the federated database. The APIs collect schema information (i.e., database entities, entity attributes, and relationships among entities) from the object-relational mapping layer provided by Django, the Python-based Model-View-Controller (MVC) development platform of choice. Thus, they are independent of the choice of the actual DataBase Management System (DBMS) adopted as a storage back-end. These metadata, stored by the DI module in its local database, will later be used to define the logical schema of the federated database (see the next section). \n2. Run SQL queries on the LAS modules\u2019 local databases and return data to the DI module. Query requests are sent by the DI module using an ad hoc, SQL-injection-proof protocol. To ensure higher flexibility and allow more complex operations not well supported by the ORM layer, the API speaks directly to the underlying DBMS using raw SQL queries and a DBMS-specific library handling the slight syntax variations among products. Currently only a library for the MySQL database management system is available. (See the subsection on query execution for further details about the query engine.)\nThe MDDM DI APIs provide a programmatic interface for executing complex queries on the federated database. The queries that may be executed through this interface are currently not free-form. Instead, they are template queries that receive zero or more parameters and return a set of rows by implementing a user-defined query flow. These template query flows may be designed by the LAS users by resorting to the Query Generator graphical user interface (GUI) (see the subsection about the query generator for details). Thus, an unlimited number of different queries may be designed and run through this interface. In addition to an endpoint for submitting query requests, the DI includes a set of APIs for obtaining the list of available templates, and a structured description of their input parameters and output schema. In this way, hard coding of result parsing routines can be avoided or at least reduced, since the data consumer can automatically detect changes in the input parameters or in the output fields and take appropriate measures. The DI APIs serve two main scenarios. First, they are used by LAS modules that need additional information from other modules to accomplish their tasks. The DI provides them with a unified, possibly aggregated set of information by running an appropriate query. Second, they may be used by external services or applications that need to query the data stored in the LAS platform (e.g., collaborations with research partners, running external pipelines or scripts).\n\nLogical design \nThe MDDM provides a logical abstraction of the databases of the different LAS modules taking part in the federated database, which may be accessed seamlessly as a single database. Initially, the MDDM DP APIs are installed in each relevant LAS module. Next, each module\u2019s database is registered in the MDDM as a DP through an ad hoc GUI, requiring the URL of the LAS module (which may reside on a different server) and a number of configuration parameters (such as a name, colors to be used by the Query Generator GUI to draw the corresponding entity buttons, etc.). The DP\u2019s schema is contextually imported through the DP API. \nBefore the DP may be queried, a number of \"entities\" and \"query paths\" must be defined by an experienced user (usually the database administrator, or an experienced PI) using a dedicated administrative GUI. Entities are classes of concepts and\/or objects that are relevant in the given context, similar to their namesakes in the Entity\u2013Relationship logical database model.[17] Within the MDDM, each entity is based on an existing table from the original DP database, but in order to enhance its expressiveness, its attributes are not limited to those included in the original DP table. Instead, any attribute from a related (under the constraint that the base table has a one-to-one or many-to-one relationship with the related table) DP table may be used or combined with other attributes to form an entity attribute. Entity attributes can be used to define filtering parameters and\/or output attributes for the entity. \nQuery paths generalize the relational notion of a foreign key, in that they dictate how two different entities A and B should be related to each other. They are defined as the set of DP tables that must be joined, through their foreign keys, in order to link the base table of entity A to the base table of entity B. Given an instance of entity A, the maximum number of instances of entity B that correspond to it can be equal to one (if the foreign key chain only includes many-to-one or one-to-one relationships) or larger than one (if it includes at least one one-to-many relationship). Query paths can be defined through an ad hoc interface. To guide the user throughout this task, the MDDM can automatically build a graph\u2014whose nodes are DP tables and edges are relationships between tables\u2014and exploit a shortest path algorithm to identify an optimal query path between a pair of entities. Alternatively, query paths can be manually built by selecting the available foreign key relationships.\n\nQuery generator \nThe MDDM\u2019s core feature is a tool to design queries. The tool exploits a graphical metaphor based on cascaded blocks that should be easily understood even by inexperienced users. Each query block represents one of the federated database entities, as previously described, or an operator, and it defines which kind of object will be returned as the block\u2019s output. Available query blocks are shown on the left-hand side of the editor (block A in Figure 2) and categorized according to the DP from which the data are drawn (e.g., the flask icon for Biobank data and the mouse icon for xenopatient data). The user can drag and drop the query blocks on the workflow editor (block B). For instance, Figure 2 shows the xenopatients, also known as patient-derived xenograft (PDX) models, and Explants blocks from the Xenopatient DP, the Aliquots block drawn from the Biobank DP, and the Containers block from the Storage DP. The reported query aims at retrieving all the containers (e.g. tubes) carrying aliquots explanted from PDX models. \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Query generator interface\n\n\n\nFiltering conditions may be specified for each block by clicking the funnel icon. In the reported example, the user can retrieve only the viable aliquots that are still available. Set operators (Union, Intersection, and Difference) and special operators (Group-count, Extend, and Template blocks), listed on the right-hand side (block C in the figure), can also be used. Blocks must be interconnected to build a query workflow by drawing a wire between a block\u2019s output terminal and another block\u2019s input terminal. Two blocks may only be connected if a query path exists (as defined by the MDDM administrator) that instructs the system how to match the corresponding entities. Conversely, an error message will be shown. Once a workflow has been defined, the user may assign a title and a description to it (block D), then either run the query immediately or save it in the system as a template, such that it can be run later either through the Query Generator Template block, or through the DI APIs. Furthermore, queries may be saved as \"translators,\" i.e., a special type of template that may be optionally run for every row appearing in the result set of a query to enrich it with additional, related information. Query results are displayed in a paged tabular structure. Since they are stored in the document database (MongoDB), they can be reopened at any time without re-running the query. In addition, they can be exported in multiple formats and saved locally by the user.\nBy using the query generator, the users can build any kind of query disregarding the structure of the databases, the query language, and the low-level procedure to connect data among several databases. For instance, they can retrieve information about treatments performed on xenopatients starting from a set of aliquots, the location of aliquots generated by cell lines, the vital status of mice belonging to a specific project, and many other data that may be valuable for daily work and for statistical analyses.\n\nQuery execution \nThe query flow designed by means of the query generator is structured as a sequence of interconnected blocks. Internally, the query is represented as a tree, whose root node is the end terminal and whose leaves are the start terminals of each initial block. Thus, queries are executed by depth-first traversing the query tree in post-order. For each visited query block, the MDDM federated database is accessed to identify the underlying DP tables. The query block is translated into a relational query structure by instantiating the required DP tables, setting the appropriate join conditions, applying the query block parameters as filtering conditions, and adding aggregation operations (if any). The choice of attributes selected from the DP tables as the block outputs depends on whether the query block is an intermediate block (thus only requiring the primary key and any correlation attributes) or the final block (requiring all attributes defined as outputs for the block). In addition, the query path linking the current block to the next one is loaded, and any other required DP tables are also instantiated. Next, the query is sent to the DP through the APIs. If both the current block and its successor reside in the same DP, the DP APIs will not issue a real query to the underlying DBMS, but they will create a logical view wrapping the query. Conversely, if the successor block belongs to a different DP, a query is issued, and result rows are returned to the MDDM. When backtracking to the successor block, the results are sent to the DP together with the new query and inserted in an indexed temporary table that is joined with the rest of the query schema. This generally improves the performance of cross-DP queries. In many cases, the genealogy ID is used as a key to link entities belonging to different DPs. As a last step, translator templates selected by the user (if any) are run on the result set. Template queries are managed by storing the tree structure of the corresponding query in the MDDM database. The tree is reloaded at each template execution, and its parameters are populated. Next, the tree is traversed as previously described.\n\nVisualization tools \nAs outlined in the introduction, translational pipelines\u2014i.e., the laboratory practice of transplanting tumor specimens into different organisms or environments to study their proliferation\u2014are of vital importance to cancer research. Therefore, practitioners need effective tools to examine the complex hierarchies of biological entities involved in the pipelines. The MDDM addresses this requirement with the Genealogy Tree Visualizer. The rationale is to leverage the intrinsic hierarchical structure of the xenopatient pipeline to produce a graphical tree representation capable of highlighting both father\u2013child relationships among xenopatients and multiple layers of relevant features for each individual by exploiting a succinct but intuitive visual encoding relying on shapes, sizes, colors, and tooltips. When the user inputs a collection identifier using a genealogy ID prefix (e.g., CRC0080), the system extracts relevant data from the MDDM federated database by exploiting dedicated templates, run through the DI APIs. Thus, by modifying such templates, the tool can be easily extended to integrate and visualize additional information. Next, xenopatients are automatically partitioned based on different features (i.e., the type of tissue from which the original human tumor was collected, as well as the lineage), and individual hierarchical trees are constructed. The result is directly shown to the user, who may customize the visualization to their liking.\nAn example is shown in Figure 3. Some of the features of each xenopatient are always visible (i.e., the shape, indicating the availability of vital material for the xenopatient, and the size, indicating the number of specimens collected from the xenopatient and available in the biobank), while others, exploiting different color coding schemas, can be layered on demand. The tool can currently visualize or highlight (i) treated, untreated, and historical (i.e., prior to the introduction of the LAS platform and, hence, not comprehensively tracked) xenopatients; (ii) experimental treatment protocols applied by practitioners; (iii) xenopatients satisfying certain treatment and biological material availability conditions; and (iv) the path from a given xenopatient to its root ancestor. In addition, statistics about treatment protocols are provided (not shown in Figure 3), whereas details about each xenopatient is displayed as a tooltip when hovering over the corresponding node. To limit the number of individuals, which can become overwhelming, the tree may be pruned by removing disconnected nodes without any treatments. Furthermore, father\u2013child relationships that are missing for historical xenopatients can be manually defined.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Genealogy tree visualizer interface\n\n\n\nPlatform usage \nThe LAS platform has been actively employed for research in the Institute for Cancer Research at Candiolo since March 2012. Only a few modules (e.g., BioBanking, Storage) were available in the first release, with a subset of the current features, while the modules addressing the molecular experiments were introduced in 2013. More recently, additional functionalities have been developed to improve the user experience and provide new functionalities to track more information about biological entities and their life cycles. Within our institution, a number of research groups are currently using the platform, and a large amount of data has already been produced and stored. In particular, as of November 2018, researchers have defined approximately 10,800 collections, each one including all biological entities (i.e., aliquots and PDXs) that share a common origin (i.e., the same collection event). The BioBanking module currently stores approximately 300,000 aliquots of different types, as pointed out in Table 1. \n\n\n\n\n\n\n\nTable 1. Aliquots statistics based on their type\n\n\n\n\nHuman\n\nXenopatients\n\nCell lines\n\n\nComplementary DNA\n\n0\n\n1414\n\n74\n\n\nComplementary RNA\n\n14\n\n1470\n\n0\n\n\nDNA\n\n17,966\n\n28,240\n\n945\n\n\nFormalin fixed\n\n1,454\n\n22,911\n\n336\n\n\nFrozen\n\n733\n\n0\n\n0\n\n\nFrozen sediment\n\n78\n\n0\n\n0\n\n\nLabeled section\n\n106\n\n3,192\n\n0\n\n\nOCT frozen\n\n1\n\n338\n\n0\n\n\nProtein\n\n0\n\n0\n\n129\n\n\nPlasma\n\n36,220\n\n185\n\n0\n\n\nParaffin section\n\n107\n\n3,478\n\n0\n\n\nRNA Later\n\n9,455\n\n56,493\n\n11\n\n\nRNA\n\n379\n\n8,449\n\n617\n\n\nSnap frozen\n\n1,453\n\n49,282\n\n134\n\n\nViable\n\n8,901\n\n30,126\n\n10,032\n\n\nTotal\n\n76,867\n\n205,578\n\n12,278\n\n\n\nOn average, 3,200 aliquots are generated each month from surgical interventions on human patients, xenopatients, cell line thawing, and derivation procedures. Since the most active user group works with PDXs, ~70% of the aliquots stored are generated from mice. Derived aliquots represent ~20% of the BioBanking module's content, and most of them are DNA. The platform also tracks aliquot consumption according to the types of experiments, as reported in Table 2. Approximately 37% are sent to external laboratories to perform special analyses, while all the other molecular experiments are performed in our institution. The in vivo experiments represent one of the main activities of our institution, as well as plasma collection. Indeed, more than 44,000 mice have been tracked. Most of them have already been explanted to generate new aliquots, while approximately 1,700 mice are currently under treatment with experimental drugs, and 1,500 are now implanted to expand the biological material. A detailed categorization of mice statuses is reported in Figure 4. At the moment of writing, more than 15,000 experimental treatments have been applied and 143 are still under execution.\n\n\n\n\n\n\n\nTable 2. Number of aliquots used for each experiment\n\n\nExperiment\n\n# of aliquots\n\n\nBeaming\n\n33\n\n\nDigitalPCR\n\n33\n\n\nHistology-IHC\n\n3,939\n\n\nMicroarray\n\n276\n\n\nRealTimePCR\n\n3,764\n\n\nSangerSequencing\n\n808\n\n\nSequenom\n\n10,508\n\n\nWesternBlots\n\n75\n\n\nCollaboration (external)\n\n11,264\n\n\n\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Mice status\n\n\n\nThe MDDM module has been widely used in the past years. We tracked more than 15,000 queries submitted by the users from the query generator. This highlights the relevance of this LAS module in the daily activities of our institution.\nThe LAS platform has been changing the research activities in our institution. We periodically ask users about their opinion on diverse system features to understand their impact on the daily work of researchers. Usually, they answer that the system is proving very useful in supporting their research, even if they met some difficulties when they started using the platform. Indeed, they admit that the LAS has changed their working habits (e.g., users of the platform must follow predefined rules and procedures for each operation and instantly report their activity). Furthermore, they noticed an improvement in the data quality and a reduction of the time spent in tracking data, especially when they work in critical environments.\nDue to the reliability of the system and the coverage of laboratory procedures, the new EurOPDX Research Infrastructure adopted LAS as a LIMS for data management, with LAS to be installed across the six nodes of the distributed infrastructure. Our lab is one of these nodes.\n\nConclusion \nThe LAS platform is designed to assist researchers of biological and biomedical laboratories in all of their activities. The modular architecture manages heterogeneous and complex data and supports researchers and practitioners in performing different experimental procedures. The graphical interfaces and the web-based architecture are suitable for use in diverse environments, including hostile ones (e.g., in sterile conditions). Due to the modular architecture, a query module has been developed to perform federated queries over all the databases. Thanks to an intuitive graphical interface, the users can design complex queries without any knowledge of databases structures or query languages. In addition, a visualization tool has been integrated in the querying environment to generate targeted visual representations of biological entities and their relationships, allowing the scientists to extract relevant information for further analyses. More visualization tools will be envisioned in the near future.\n\nAcknowledgements \nThe EurOPDX Research Infrastructure has received funding from the European Union's Horizon 2020 research and innovation program, grant agreement no. #731105 (EDIReX project). A.B. has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No 724748) and AIRC (IG n 20697 and Special Program 5 per mille metastases Project n 21091).\n\nSystem availability \nLAS can be found on GitHub.\n\nConflict of interest \nNone declared.\n\nReferences \n\n\n\u2191 Chen, Y.; Lin, Y.; Yuan, X. et al.. \"Chapter 9: LIMS and Clinical Data Management\". In Shen, B.; Tang, H.; Jiang, X.. Translational Biomedical Informatics: A Precision Medicine Perspective. Springer. pp. 225\u2013240. doi:10.1007\/978-981-10-1503-8_9. ISBN 9789811015038.   \n\n\u2191 Wood, S. (September 2007). \"Comprehensive Laboratory Informatics: A Multilayer Approach\" (PDF). American Laboratory. pp. 3. Archived from the original on 25 August 2017. https:\/\/web.archive.org\/web\/20170825181932\/https:\/\/www.it.uu.se\/edu\/course\/homepage\/lims\/vt12\/ComprehensiveLaboratoryInformatics.pdf .   \n\n\u2191 \"Starlims\". Abbott. 2018. https:\/\/www.informatics.abbott\/us\/en\/offerings\/lims .   \n\n\u2191 \"Sapio Sciences\". Sapio Sciences, LLC. 2018. https:\/\/www.sapiosciences.com\/ .   \n\n\u2191 \"LabVantage\". LabVantage Solutions, Inc. 2018. https:\/\/www.labvantage.com\/ .   \n\n\u2191 Blankenberg, D.; Von Kuster, G.; Coraor, N. et al. (2010). \"Galaxy: a web-based genome analysis tool for experimentalists\". Current Protocols in Molecular Biology 19 (Unit 19.10.1\u201321). doi:10.1002\/0471142727.mb1910s89. PMC PMC4264107. PMID 20069535. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4264107 .   \n\n\u2191 Goecks, J.; Nekrutenko, A.; Taylor, J.; The Galaxy Team (2010). \"Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences\". Genome Biology 11 (8): R86. doi:10.1186\/gb-2010-11-8-r86. PMC PMC2945788. PMID 20738864. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2945788 .   \n\n\u2191 O'Connor, B.D.; Merriman, B.; Nelson, S.F. (2010). \"SeqWare Query Engine: storing and searching sequence data in the cloud\". BMC Bioinformatics 11 (Suppl. 12): S2. doi:10.1186\/1471-2105-11-S12-S2. http:\/\/www.biomedcentral.com\/1471-2105\/11\/S12\/S2 .   \n\n\u2191 Baralis, E.; Bertotti, A.; Fiori, A.; Grand, A. (2012). \"LAS: A software platform to support oncological data management\". Journal of Medical System 36 (Suppl. 1): S81\u201390. doi:10.1007\/s10916-012-9891-6. PMID 23117791.   \n\n\u2191 Forbes, S.A.; Beare, D.; Gunasekaran, P. et al. (2015). \"COSMIC: Exploring the world's knowledge of somatic mutations in human cancer\". Nucleic Acids Research 43 (DB1): D805\u201311. doi:10.1093\/nar\/gku1075. PMC PMC4383913. PMID 25355519. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383913 .   \n\n\u2191 Forbes, S.A.; Bindal, N.; Beare, D. et al. (2016). \"Abstract 5285: COSMIC: Comprehensively exploring oncogenomics\". Cancer Research 76 (14 Suppl.): 5285. doi:10.1158\/1538-7445.AM2016-5285.   \n\n\u2191 Kitts, A.; Phan, L.; Ward, M. et al. (2013). \"The Database of Short Genetic Variation (dbSNP)\". The NCBI Handbook (2nd ed.). National Center for Biotechnology Information. https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK174586\/ .   \n\n\u2191 Eilbeck, K.; Lewis, S.E.; Mungall, C.J. et al. (2005). \"The Sequence Ontology: A tool for the unification of genome annotations\". Genome Biology 6 (5): R44. doi:10.1186\/gb-2005-6-5-r44. PMC PMC1175956. PMID 15892872. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1175956 .   \n\n\u2191 Vukotic, A.; Watt, N.; Abedrabbo, T. et al. (2014). Neo4j in Action. Manning Publications. ISBN 9781617290763.   \n\n\u2191 Bertotti, A.; Migliardi, G.; Galimi, F. et al. (2011). \"A molecularly annotated platform of patient-derived xenografts (\"xenopatients\") identifies HER2 as an effective therapeutic target in cetuximab-resistant colorectal cancer\". Cancer Discovery 1 (6): 508\u201323. doi:10.1158\/2159-8290.CD-11-0109. PMID 22586653.   \n\n\u2191 Sheth, A.P.; Larson, J.A. (1990). \"Federated database systems for managing distributed, heterogeneous, and autonomous databases\". ACM Computing Surveys 22 (3): 183\u2013236. doi:10.1145\/96602.96604.   \n\n\u2191 Atzeni, P.; Ceri, S.; Paraboschi, S.; Torlone, R. (1999). Database Systems: Concepts, Languages & Architectures. The McGraw-Hill Companies. ISBN 0077095006.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\">https:\/\/www.limswiki.org\/index.php\/Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 21 January 2020, at 00:49.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 466 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","45b5e684bad1c44cd9a1cb43bb6b311e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_One_tool_to_find_them_all_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:One tool to find them all: A case of data integration and querying in a distributed LIMS platform<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>In recent years, <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> (LIMS) have been growing from mere inventory systems into increasingly comprehensive software platforms, spanning functionalities as diverse as data search, annotation, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analysis<\/a>. In 2011, our institution started a LIMS project named the Laboratory Assistant Suite with the purpose of assisting researchers throughout all of their <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> activities, providing graphical tools to support decision-making tasks and building complex analyses on integrated data. The modular architecture of the system exploits multiple databases with different technologies. To provide an efficient and easy tool for retrieving information of interest, we developed the Multi-Dimensional Data Manager (MDDM). By means of intuitive interfaces, scientists can execute complex queries without any knowledge of query languages or database structures, and easily integrate heterogeneous data stored in multiple databases. Together with the other software modules making up the platform, the MDDM has helped improve the overall quality of the data, substantially reduced the time spent with manual data entry and retrieval, and ultimately broadened the spectrum of interconnections among the data, offering novel perspectives to biomedical analysts.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>The introduction of automation and high-throughput technologies in <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> environments has raised diverse issues related to the amount and heterogeneity of the data produced, the adoption of robust procedures for <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">sample<\/a> tracking, and the management of computer-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a> needed to process and analyze the raw data. <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">Laboratory information management systems<\/a> (LIMS) have gained increasing popularity because they can ensure good levels of <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> over laboratory activities and efficiently handle the large amounts of data produced.<sup id=\"rdp-ebb-cite_ref-ChenLIMS16_1-0\" class=\"reference\"><a href=\"#cite_note-ChenLIMS16-1\">[1]<\/a><\/sup>\n<\/p><p>LIMS aim at assisting the researchers in their daily laboratory practice, improving the accessibility of instruments, and tracking biological samples and their related <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>.\n<\/p><p>In the past decade, several open-source as well as proprietary LIMS have been developed. Commercial solutions are typically large, complex, and feature-rich products designed to easily support large laboratories. Their license fees can be prohibitive, and extra features may come at additional costs.<sup id=\"rdp-ebb-cite_ref-CLIPaper_2-0\" class=\"reference\"><a href=\"#cite_note-CLIPaper-2\">[2]<\/a><\/sup> To reduce these costs, the last generation of commercial LIMS adopt web-oriented software technologies, particularly the <a href=\"https:\/\/www.limswiki.org\/index.php\/Software_as_a_service\" title=\"Software as a service\" class=\"wiki-link\" data-key=\"ae8c8a7cd5ee1a264f4f0bbd4a4caedd\">software-as-a-service<\/a> distribution model, which reduces the customer\u2019s final expenditure on license fees, hardware, and maintenance. Examples of commercial solutions include <a href=\"https:\/\/www.limswiki.org\/index.php\/Abbott_Informatics_Corporation\" title=\"Abbott Informatics Corporation\" class=\"wiki-link\" data-key=\"90479bab12020c96864541696a62156f\">STARLIMS<\/a><sup id=\"rdp-ebb-cite_ref-AbbottStarlims18_3-0\" class=\"reference\"><a href=\"#cite_note-AbbottStarlims18-3\">[3]<\/a><\/sup>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Sapio_Sciences,_LLC\" title=\"Sapio Sciences, LLC\" class=\"wiki-link\" data-key=\"c013f3c0c7c61713ca1d19812d6aac24\">Exemplar LIMS<\/a><sup id=\"rdp-ebb-cite_ref-SapioSapio18_4-0\" class=\"reference\"><a href=\"#cite_note-SapioSapio18-4\">[4]<\/a><\/sup>, and <a href=\"https:\/\/www.limswiki.org\/index.php\/LabVantage_Solutions,_Inc.\" title=\"LabVantage Solutions, Inc.\" class=\"wiki-link\" data-key=\"1855d5158d4dd8f748e925e626daf5ee\">LabVantage<\/a>.<sup id=\"rdp-ebb-cite_ref-LabVantageLabVantage18_5-0\" class=\"reference\"><a href=\"#cite_note-LabVantageLabVantage18-5\">[5]<\/a><\/sup>\n<\/p><p>Commercial LIMS tend to offer features based on common laboratory procedures and best practices, which may not fit highly specific settings well. For instance, LabVantage provides a large set of features, such as sample and batch management, quality control, advanced storage and logistics, and task scheduling. However, the life cycle of xenopatients (i.e., biological models for cancer research based on the transplantation of human tumors in mice) is not available in the standard software and should be implemented as a custom module by the software developer. Another issue that affects commercial LIMSs is the management and standardization of <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomic<\/a> data. To the best of our knowledge, these systems do not exploit any knowledge base related to the genomic data and do not provide any validation and analysis of different genomic data stored in the system.\n<\/p><p>Other open-source solutions like <a href=\"https:\/\/www.limswiki.org\/index.php\/Galaxy_(biomedical_software)\" title=\"Galaxy (biomedical software)\" class=\"wiki-link\" data-key=\"ead5d6ebaa8d67744d2f68d454d89ce6\">Galaxy<\/a><sup id=\"rdp-ebb-cite_ref-BlankenbergGalaxy10_6-0\" class=\"reference\"><a href=\"#cite_note-BlankenbergGalaxy10-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GoecksGalaxy10_7-0\" class=\"reference\"><a href=\"#cite_note-GoecksGalaxy10-7\">[7]<\/a><\/sup> focus instead on specific sub-domains, addressing DNA sequencing and annotation, or <a href=\"https:\/\/www.limswiki.org\/index.php\/SeqWare\" title=\"SeqWare\" class=\"wiki-link\" data-key=\"da4ce5a823f703b673f1ef4b38860d9a\">SeqWare<\/a><sup id=\"rdp-ebb-cite_ref-O.27ConnorSeqWare10_8-0\" class=\"reference\"><a href=\"#cite_note-O.27ConnorSeqWare10-8\">[8]<\/a><\/sup>, which tracks <i>in vivo<\/i> and <i>in vitro<\/i> experiments and allows for complex analysis workflows.\n<\/p><p>For this reason, many institutions have invested in the development of in-house solutions and\/or have adapted open-source projects to their own requirements. In this way, the developed solutions can provide functionality that meet the specific needs of the researchers in their institution laboratories. From an engineering perspective, developing in-house solutions may also permit the exploration and adoption of new technologies, in order to define better data models and improve system performance.\n<\/p><p>To address a substantial mismatch between the LIMS solutions on offer and the functional requirements dictated by research practice, the Institute for Cancer Research at Candiolo (Italy) started to implement its own LIMS, named the Laboratory Assistant Suite (LAS) platform, in 2011.<sup id=\"rdp-ebb-cite_ref-BaralisLAS12_9-0\" class=\"reference\"><a href=\"#cite_note-BaralisLAS12-9\">[9]<\/a><\/sup> The main purpose of the platform was to assist researchers in different laboratory and research activities, allowing management of different kinds of raw data (e.g., biological, molecular), tracking experimental data, supporting decision-making tasks, and integrating heterogeneous data for complex analyses. As development progressed, several new features and modules were included to (i) track clinical data, (ii) include support to the newest technologies exploited for molecular experiments, and (iii) standardize the description of genomic data by means of semantic web technologies. Thanks to these new features, scientists can gain better insight into tumor development by jointly studying the clinical evolution of the disease and the experimental results derived from <i>in vivo<\/i> and <i>in vitro<\/i> experimentation. The experimental pipelines exploited in the translational research context are the primary focus of the LAS, which targets the standardization of the genomic data to allow a comparison of results coming from different technologies.\n<\/p><p>Still, unlike the other commercial and open-source platforms, the LAS makes an attempt at covering a wide range of diverse laboratory procedures and, thanks to its versatile and general-purpose structure, it can be extended to support new ones with limited effort.\n<\/p><p>Thanks to the vast variety of different experimental technologies supported by the LAS and their high level of specificity, large amounts of heterogeneous and complex information are collected in separate databases. To enable the users to extract and correlate information from the different databases exploited by the platform, a Multi-Dimensional Data Manager (MDMM) module was developed. The module takes care of merging data from the different LAS databases and provides a simple graphical user interface to extract the information of interest without any knowledge of a query language. A tool to visualize biological entities and their related information with a hierarchical tree structure is also available, while other powerful visualization tools are currently under development. To the best of our knowledge, no similar tools applied to biological data and distributed databases exist.\n<\/p><p>This paper presents the main characteristics of the LAS and its exploitation in the research laboratories of the Institute for Cancer Research at Candiolo by its researchers and research partners. Next, the main features of the MDDM are described. Afterwards, current and future research directions are presented.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"LAS\">LAS<\/span><\/h2>\n<p>Translational research aims at enhancing patient care and transferring scientific discoveries from the laboratory to a real clinical context. It is a kind of metaphorical scientific cycle from bench to bedside and back again through complex iterative processes, operating between laboratory (i.e., preclinical research) and clinic. To the aim of managing and integrating preclinical and clinical information, a robust but flexible data management platform is needed. In particular, different types of information (e.g., biological data, molecular data, procedure tracking data, and sample tracking data)\u2014some of which can be highly complex\u2014should be independently managed by the platform but, at the same time, interconnected to permit integrated analyses.\n<\/p><p>The LAS platform is freely available upon request to the authors. The software is distributed by means of a Docker-based approach to allow interested organizations to configure it according to their constraints. Moreover, the use of Docker allows system administrators to run the software on different servers using the Docker Swarm configuration for balancing the workload, as well as the associated data resources. We usually recommend installing the LAS on at least two servers, one dedicated to the containers running the software and the other for databases. The servers characteristics depend on several aspects, such as the number of simultaneously logged users, the number of biological entities tracked, and the dimension of raw data stored. We suggest as initial setup a server with at least 16 GB of RAM and a storage space of 2 TB. Interested users may refer to the video tutorials (available at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/lasrep.ircc.it\/\" target=\"_blank\">http:\/\/lasrep.ircc.it\/<\/a>) to explore main system features and as a reference guide during its usage. The documentation of the platform is provided with the software and can be downloaded from the documentation section of the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/las.ircc.it\/\" target=\"_blank\">LAS instance<\/a> of the Institute for Cancer Research at Candiolo.\n<\/p><p>What follows is a description of the data architecture and the main functionalities included in the platform.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_models\">Data models<\/span><\/h3>\n<p>The LAS platform has been developed using different database technologies to fit the needs of the application, and to handle in a suitable way the heterogeneous data characteristics.\n<\/p><p>The platform makes use of relational databases to track biological entities and their properties, as well as the information about the various experimental procedures. Since the platform includes different modules managing substantially different types of entities and\/or specific laboratory procedures, different database instances are exploited. The core biological entities (i.e., Aliquot, Biomouse, and Cell Line) are identified by a unique and mnemonic key named GenealogyID that encodes relevant information regarding the history of the entity. This key is automatically generated by the LAS platform through formal rules and may be used to link the data across the databases.\n<\/p><p>Parallel to the relational databases storing operating data, a graph database is exploited. It is used to represent the complex inherent hierarchy of biological entities and their relationships. Being able to easily and efficiently reconstruct the genealogical tree of each entity is indeed an essential feature of the platform, allowing the user to perform <i>ad hoc<\/i> queries and to isolate specific sub-trees of biological entities involved in the experimental pipeline. Moreover, the graph database has been exploited to store a knowledge base for the heterogeneous domains managed by the LAS modules. By using a graph representation, all these domains can be easily interconnected, while the knowledge base can be continuously updated and augmented with new layers of information and different levels of abstraction (e.g., proteomics, clinical, etc.). Finally, a social network of users and research groups using the LAS platform is also stored in the graph, to model data ownership, resolve data access conflicts, and manage data sharing and collaboration among different groups or users.\n<\/p><p>A document database, MongoDB, is also used to store files associated with biological entities and metadata generated by both the LAS Genomic Data Manager modules and the MDDM. The latter usage will be discussed in detail in the next section.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Functionalities\">Functionalities<\/span><\/h3>\n<p>The LAS architecture includes a set of software modules, i.e., fully-fledged web applications, each addressing a different type of biological entity and its associated experimental procedures. Modules may interact with each other by means of web <a href=\"https:\/\/www.limswiki.org\/index.php\/Application_programming_interface\" title=\"Application programming interface\" class=\"wiki-link\" data-key=\"36fc319869eba4613cb0854b421b0934\">application programming interfaces<\/a> (APIs), e.g., to exchange data and\/or to carry out operations that span multiple entities or domains. (The modules currently included in the LAS platform are described in the following.)\n<\/p><p>Even if the platform has been developed since 2011, we always took into account security issues during the design and development processes of the platform. In this way, our software is compliant with the constraints of the <a href=\"https:\/\/www.limswiki.org\/index.php\/General_Data_Protection_Regulation\" title=\"General Data Protection Regulation\" class=\"wiki-link\" data-key=\"3f4bdf6f0dcb360b1e79aad8674c2447\">General Data Protection Regulation<\/a> (GDPR), with enforcement beginning on May 25, 2018. Indeed, the management of data produced by different users and\/or groups requires that the access to functionalities and information are restricted according to several criteria such as group and\/or project membership, and user role. For these reasons, the platform manages users and their privileges following these concepts:\n<\/p>\n<ul><li> Working Group: A Working Group (WG) is a set of users in the LAS platform that work together toward a specific goal (e.g., project, research activity). The data produced by the users of the same group are private, unless they intentionally share data with other groups.<\/li><\/ul>\n<ul><li> User Profiles: Each user belonging to a WG has a set of permissions to access the LAS functionalities they have been enabled to use. These functionalities are defined according to the role selected during the user registration process. The manager of their WG or the system administrator can assign new functionalities upon request.<\/li><\/ul>\n<p>To collect data, the user is required to specify the informed consent signed by the patient for specific research activities (e.g., preclinical trials) involving personal samples and information. This document is defined by a committee to accomplish all the constraints included in the GDPR. Since the data are collected for research purposes, the patient can only revoke the usage of the biological samples, but not the information (e.g., experimental results) collected by the researchers. Only the researchers that are included in the research project can manage these samples and track the experimental processes according to their profile. The platform tracks all the procedures performed by each user in order to identify malicious usage of the software.\n<\/p><p>For each patient, the Clinical module tracks both contextual information (i.e., personal data, Medical Center of the Trial, etc.) and relevant clinical events through a case report form. All data are related to the relative informed consent that grants data and specimen sampling.\n<\/p><p>The BioBanking Management module covers a wide range of activities, including management of biological samples and associated pathological information, as well as support for a number of laboratory-related procedures. For instance, the module can handle the collection of biological material from surgical intervention and the acquisition of aliquots from external laboratories. Aliquots stored in the system are characterized by features such as tumor type (e.g., colorectal), tissue type (e.g., liver metastasis), source <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a>, or laboratory and pathological information. Measurement of aliquot physical characteristics such as volume, concentration, purity, and quality can be tracked by the module, as well as the derivation of new biological materials (e.g., DNA and cDNA) and the planning of molecular experiments.\n<\/p><p>The biological material used in our laboratories is stored by means of several types of containers (e.g., freezers, racks, plates, and tubes). Their mutual interactions (i.e., which types of containers can host other containers) can change according to characteristics such as the layout and the laboratory procedure. Additionally, the Storage Management module allows managing any kind of container by defining and applying different rules to them, and it tracks the relationships between the containers and the biological entities.\n<\/p><p>Different types of molecular analyses can be conducted on biological samples, to investigate various aspects of their genetic constituents that may have an impact on the development of oncogenic behavior. For instance, biologists may be interested in analyzing mutations for a target gene involved in tumor proliferation. In an effort to closely track the translational research pipeline from the collection of samples to their analysis, the LAS provides support to tracking the most frequently used molecular profiling techniques in our institution {e.g., Sanger sequencing, real-time polymerase chain reaction (PCR), and Sequenom). Each molecular module queries the knowledge base of the Genomic Annotation Manager (GAM) to retrieve the description of its reagents, as well as a specification of all possible alterations (e.g., sequence alterations and gene copy number variations) known in the literature, to allow both the experiment definition and the evaluation of experimental results.\n<\/p><p>The GAM provides a higher-level, qualitative insight into the genomic features of biological samples. This information is shaped in the form of annotations, i.e., a set of semantic labels attached to a sample, pointing out some of its relevant features. To ensure semantic coherence and adopt a standardized nomenclature, all relevant concepts from the genomic and biological domains used for labeling samples have been drawn from a number of public, freely accessible databases and ontologies.<sup id=\"rdp-ebb-cite_ref-ForbesCOSMIC15_10-0\" class=\"reference\"><a href=\"#cite_note-ForbesCOSMIC15-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ForbesAbstract16_11-0\" class=\"reference\"><a href=\"#cite_note-ForbesAbstract16-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KittsTheData13_12-0\" class=\"reference\"><a href=\"#cite_note-KittsTheData13-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-EilbeckTheSeq05_13-0\" class=\"reference\"><a href=\"#cite_note-EilbeckTheSeq05-13\">[13]<\/a><\/sup> This information has been structured into a knowledge base, modeled as a graph, and stored in a graph database.<sup id=\"rdp-ebb-cite_ref-VukoticNeo4j14_14-0\" class=\"reference\"><a href=\"#cite_note-VukoticNeo4j14-14\">[14]<\/a><\/sup> Concepts are interlinked with one another according to both general-purpose semantic relationships such as containment (\"part of\") or generalization (\"is a\"), and domain-specific relationships (e.g., indicating an underlying biochemical process, as in \"is transcribed from\"). New concepts and relationships, as well as new domains of interest, may be added or layered as needed to account for novel findings and broaden the spectrum of investigation. Within the GAM, every annotation is a semantic statement establishing a relationship, expressed by means of a predicate, between a biological sample (the subject of the statement) and a concept (the object of the statement), such as a genetic mutation. It is represented within the graph database as a node of type \"annotation\" with a pair of incoming and outgoing edges, one linking the biological sample to the annotation node by means of a <tt>has_annotation<\/tt> relationship, and the other linking the annotation node to the reference node in the knowledge base by means of a <tt>has_reference<\/tt> relationship. The annotation node is often linked to other nodes, such as the process that produced the annotation or the raw experimental data.\n<\/p><p>Finally, the platform allows tracking <i>in vivo<\/i> and <i>in vitro<\/i> experiments. Studies that are \"<i>in vivo<\/i>\" (Latin for \"within the living\") are those in which the effects of various biological entities are tested on whole, living organisms, usually animals (e.g., xenopatients). This kind of experiment is normally exploited to test drug therapies and expand the collection of biological samples. We based our development on the model described by Bertotti <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BertottiAMolec11_15-0\" class=\"reference\"><a href=\"#cite_note-BertottiAMolec11-15\">[15]<\/a><\/sup> to manage immunocompromised animals and monitor the xenopatient life cycle, from their acquisition by the research institute to their death. Unlike <i>in vivo<\/i> experiments, <i>in vitro<\/i> studies are performed with cells or biological molecules studied outside their normal biological context. The LAS platform defines a cell line as the set of biological entities that are generated from the same biological entity and are under the same experimental conditions. The experimental conditions are defined by the protocols that describe the type of process (i.e., adherent, suspend, and organoid) and the set of culturing conditions applied (e.g., nutrients and chemicals, hormones\/growth factors, and antibiotics). In addition, the platform allows the management of the generation and thawing procedures of cell lines.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"MDDM\">MDDM<\/span><\/h2>\n<p>Besides tracking experimental procedures and recording all the data related to biological entities, the retrieval of the information tracked by the platform is fundamental to discover new knowledge related to tumors. Due to the large number of laboratory activities and biological entities managed by the LAS with a high level of specificity, and to streamline the development and maintenance process over time, each LAS module operates on a separate database instance, storing both operational data and domain-specific knowledge, and partially replicating key information from other LAS modules. As a result, while each instance focuses on its own specific subset of functionalities, no one database can provide a comprehensive insight into the data. Nonetheless, building actionable knowledge requires the integration of heterogeneous information to establish connections among different and related biological entities, or to explore different facets of the same entities, both for operational and for research purposes. For instance, identifying the number of tubes stored in each plate of a given freezer and their content may be required for managing physical storage equipment. From a researcher\u2019s perspective, the study of the evolution of the tumor mass in a mouse under pharmacological treatment (monitored by means of <i>in vivo<\/i> experimentation and tracked by the Xenograft Management module) must be traced back to the originating patient (managed by the Clinical module), and should be further correlated to the mouse genomic mutational status and gene expression levels (collected by the set of Molecular modules and annotated by the Genomic Annotation module). The MDDM addresses the issue of extracting all information of interest stored in the databases of each LAS module, providing the end user with an intuitive graphical tool for building customized queries, with a unified view on the entire collection of databases. In addition, the MDDM can be exploited to run programmatic queries via a set of web APIs, enabling other LAS modules, as well as other external applications, to retrieve data of interest.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Architecture\">Architecture<\/span><\/h3>\n<p>The MDDM has a distributed architecture. Its main components, shown in Figure 1, are the central MDDM Data Integrator (DI) module and the distributed MDDM Data Provider (DP) modules. The DI module provides a unified logical view on the databases of the other modules, thus acting as a (read-only) federated database.<sup id=\"rdp-ebb-cite_ref-ShethFed90_16-0\" class=\"reference\"><a href=\"#cite_note-ShethFed90-16\">[16]<\/a><\/sup> The DP modules rely on a standalone application layer that must be installed within each LAS module, allowing the LAS modules to join the federated database. Interactions between the DI and the DP modules occur via two distinct sets of APIs.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Grand_Database2019_2019.png\" class=\"image wiki-link\" data-key=\"726c87d69c55a3c0ea467d3376097d43\"><img alt=\"Fig1 Grand Database2019 2019.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/bd\/Fig1_Grand_Database2019_2019.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> MDMM architecture<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The MDDM DP APIs are called by the central DI module. They provide an interface to perform the following tasks:\n<\/p><p>1. Retrieve schema information from each LAS module\u2019s database. This operation is executed once at setup time, when the LAS module is initially registered as a member of the federated database. The APIs collect schema information (i.e., database entities, entity attributes, and relationships among entities) from the object-relational mapping layer provided by Django, the Python-based Model-View-Controller (MVC) development platform of choice. Thus, they are independent of the choice of the actual DataBase Management System (DBMS) adopted as a storage back-end. These metadata, stored by the DI module in its local database, will later be used to define the logical schema of the federated database (see the next section). \n<\/p><p>2. Run SQL queries on the LAS modules\u2019 local databases and return data to the DI module. Query requests are sent by the DI module using an <i>ad hoc<\/i>, SQL-injection-proof protocol. To ensure higher flexibility and allow more complex operations not well supported by the ORM layer, the API speaks directly to the underlying DBMS using raw SQL queries and a DBMS-specific library handling the slight syntax variations among products. Currently only a library for the <a href=\"https:\/\/www.limswiki.org\/index.php\/MySQL\" title=\"MySQL\" class=\"wiki-link\" data-key=\"35005451bfcd508bce47c58e72260128\">MySQL<\/a> database management system is available. (See the subsection on query execution for further details about the query engine.)\n<\/p><p>The MDDM DI APIs provide a programmatic interface for executing complex queries on the federated database. The queries that may be executed through this interface are currently not free-form. Instead, they are template queries that receive zero or more parameters and return a set of rows by implementing a user-defined query flow. These template query flows may be designed by the LAS users by resorting to the Query Generator graphical user interface (GUI) (see the subsection about the query generator for details). Thus, an unlimited number of different queries may be designed and run through this interface. In addition to an endpoint for submitting query requests, the DI includes a set of APIs for obtaining the list of available templates, and a structured description of their input parameters and output schema. In this way, hard coding of result parsing routines can be avoided or at least reduced, since the data consumer can automatically detect changes in the input parameters or in the output fields and take appropriate measures. The DI APIs serve two main scenarios. First, they are used by LAS modules that need additional information from other modules to accomplish their tasks. The DI provides them with a unified, possibly aggregated set of information by running an appropriate query. Second, they may be used by external services or applications that need to query the data stored in the LAS platform (e.g., collaborations with research partners, running external pipelines or scripts).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Logical_design\">Logical design<\/span><\/h3>\n<p>The MDDM provides a logical abstraction of the databases of the different LAS modules taking part in the federated database, which may be accessed seamlessly as a single database. Initially, the MDDM DP APIs are installed in each relevant LAS module. Next, each module\u2019s database is registered in the MDDM as a DP through an <i>ad hoc<\/i> GUI, requiring the URL of the LAS module (which may reside on a different server) and a number of configuration parameters (such as a name, colors to be used by the Query Generator GUI to draw the corresponding entity buttons, etc.). The DP\u2019s schema is contextually imported through the DP API. \n<\/p><p>Before the DP may be queried, a number of \"entities\" and \"query paths\" must be defined by an experienced user (usually the database administrator, or an experienced PI) using a dedicated administrative GUI. Entities are classes of concepts and\/or objects that are relevant in the given context, similar to their namesakes in the Entity\u2013Relationship logical database model.<sup id=\"rdp-ebb-cite_ref-AtzeniData99_17-0\" class=\"reference\"><a href=\"#cite_note-AtzeniData99-17\">[17]<\/a><\/sup> Within the MDDM, each entity is based on an existing table from the original DP database, but in order to enhance its expressiveness, its attributes are not limited to those included in the original DP table. Instead, any attribute from a related (under the constraint that the base table has a one-to-one or many-to-one relationship with the related table) DP table may be used or combined with other attributes to form an entity attribute. Entity attributes can be used to define filtering parameters and\/or output attributes for the entity. \n<\/p><p>Query paths generalize the relational notion of a foreign key, in that they dictate how two different entities A and B should be related to each other. They are defined as the set of DP tables that must be joined, through their foreign keys, in order to link the base table of entity A to the base table of entity B. Given an instance of entity A, the maximum number of instances of entity B that correspond to it can be equal to one (if the foreign key chain only includes many-to-one or one-to-one relationships) or larger than one (if it includes at least one one-to-many relationship). Query paths can be defined through an <i>ad hoc<\/i> interface. To guide the user throughout this task, the MDDM can automatically build a graph\u2014whose nodes are DP tables and edges are relationships between tables\u2014and exploit a shortest path algorithm to identify an optimal query path between a pair of entities. Alternatively, query paths can be manually built by selecting the available foreign key relationships.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Query_generator\">Query generator<\/span><\/h3>\n<p>The MDDM\u2019s core feature is a tool to design queries. The tool exploits a graphical metaphor based on cascaded blocks that should be easily understood even by inexperienced users. Each query block represents one of the federated database entities, as previously described, or an operator, and it defines which kind of object will be returned as the block\u2019s output. Available query blocks are shown on the left-hand side of the editor (block A in Figure 2) and categorized according to the DP from which the data are drawn (e.g., the flask icon for Biobank data and the mouse icon for xenopatient data). The user can drag and drop the query blocks on the workflow editor (block B). For instance, Figure 2 shows the xenopatients, also known as patient-derived xenograft (PDX) models, and Explants blocks from the Xenopatient DP, the Aliquots block drawn from the Biobank DP, and the Containers block from the Storage DP. The reported query aims at retrieving all the containers (e.g. tubes) carrying aliquots explanted from PDX models. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Grand_Database2019_2019.png\" class=\"image wiki-link\" data-key=\"4ab8882c47d2dd775b7799d83cbf4fa6\"><img alt=\"Fig2 Grand Database2019 2019.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a3\/Fig2_Grand_Database2019_2019.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Query generator interface<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Filtering conditions may be specified for each block by clicking the funnel icon. In the reported example, the user can retrieve only the viable aliquots that are still available. Set operators (Union, Intersection, and Difference) and special operators (Group-count, Extend, and Template blocks), listed on the right-hand side (block C in the figure), can also be used. Blocks must be interconnected to build a query workflow by drawing a wire between a block\u2019s output terminal and another block\u2019s input terminal. Two blocks may only be connected if a query path exists (as defined by the MDDM administrator) that instructs the system how to match the corresponding entities. Conversely, an error message will be shown. Once a workflow has been defined, the user may assign a title and a description to it (block D), then either run the query immediately or save it in the system as a template, such that it can be run later either through the Query Generator Template block, or through the DI APIs. Furthermore, queries may be saved as \"translators,\" i.e., a special type of template that may be optionally run for every row appearing in the result set of a query to enrich it with additional, related information. Query results are displayed in a paged tabular structure. Since they are stored in the document database (MongoDB), they can be reopened at any time without re-running the query. In addition, they can be exported in multiple formats and saved locally by the user.\n<\/p><p>By using the query generator, the users can build any kind of query disregarding the structure of the databases, the query language, and the low-level procedure to connect data among several databases. For instance, they can retrieve information about treatments performed on xenopatients starting from a set of aliquots, the location of aliquots generated by cell lines, the vital status of mice belonging to a specific project, and many other data that may be valuable for daily work and for statistical analyses.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Query_execution\">Query execution<\/span><\/h3>\n<p>The query flow designed by means of the query generator is structured as a sequence of interconnected blocks. Internally, the query is represented as a tree, whose root node is the end terminal and whose leaves are the start terminals of each initial block. Thus, queries are executed by depth-first traversing the query tree in post-order. For each visited query block, the MDDM federated database is accessed to identify the underlying DP tables. The query block is translated into a relational query structure by instantiating the required DP tables, setting the appropriate join conditions, applying the query block parameters as filtering conditions, and adding aggregation operations (if any). The choice of attributes selected from the DP tables as the block outputs depends on whether the query block is an intermediate block (thus only requiring the primary key and any correlation attributes) or the final block (requiring all attributes defined as outputs for the block). In addition, the query path linking the current block to the next one is loaded, and any other required DP tables are also instantiated. Next, the query is sent to the DP through the APIs. If both the current block and its successor reside in the same DP, the DP APIs will not issue a real query to the underlying DBMS, but they will create a logical view wrapping the query. Conversely, if the successor block belongs to a different DP, a query is issued, and result rows are returned to the MDDM. When backtracking to the successor block, the results are sent to the DP together with the new query and inserted in an indexed temporary table that is joined with the rest of the query schema. This generally improves the performance of cross-DP queries. In many cases, the genealogy ID is used as a key to link entities belonging to different DPs. As a last step, translator templates selected by the user (if any) are run on the result set. Template queries are managed by storing the tree structure of the corresponding query in the MDDM database. The tree is reloaded at each template execution, and its parameters are populated. Next, the tree is traversed as previously described.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Visualization_tools\">Visualization tools<\/span><\/h3>\n<p>As outlined in the introduction, translational pipelines\u2014i.e., the laboratory practice of transplanting tumor specimens into different organisms or environments to study their proliferation\u2014are of vital importance to cancer research. Therefore, practitioners need effective tools to examine the complex hierarchies of biological entities involved in the pipelines. The MDDM addresses this requirement with the Genealogy Tree Visualizer. The rationale is to leverage the intrinsic hierarchical structure of the xenopatient pipeline to produce a graphical tree representation capable of highlighting both father\u2013child relationships among xenopatients and multiple layers of relevant features for each individual by exploiting a succinct but intuitive visual encoding relying on shapes, sizes, colors, and tooltips. When the user inputs a collection identifier using a genealogy ID prefix (e.g., CRC0080), the system extracts relevant data from the MDDM federated database by exploiting dedicated templates, run through the DI APIs. Thus, by modifying such templates, the tool can be easily extended to integrate and visualize additional information. Next, xenopatients are automatically partitioned based on different features (i.e., the type of tissue from which the original human tumor was collected, as well as the lineage), and individual hierarchical trees are constructed. The result is directly shown to the user, who may customize the visualization to their liking.\n<\/p><p>An example is shown in Figure 3. Some of the features of each xenopatient are always visible (i.e., the shape, indicating the availability of vital material for the xenopatient, and the size, indicating the number of specimens collected from the xenopatient and available in the biobank), while others, exploiting different color coding schemas, can be layered on demand. The tool can currently visualize or highlight (i) treated, untreated, and historical (i.e., prior to the introduction of the LAS platform and, hence, not comprehensively tracked) xenopatients; (ii) experimental treatment protocols applied by practitioners; (iii) xenopatients satisfying certain treatment and biological material availability conditions; and (iv) the path from a given xenopatient to its root ancestor. In addition, statistics about treatment protocols are provided (not shown in Figure 3), whereas details about each xenopatient is displayed as a tooltip when hovering over the corresponding node. To limit the number of individuals, which can become overwhelming, the tree may be pruned by removing disconnected nodes without any treatments. Furthermore, father\u2013child relationships that are missing for historical xenopatients can be manually defined.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Grand_Database2019_2019.png\" class=\"image wiki-link\" data-key=\"eb00503ef296ca84bd9b43b5fe7ff766\"><img alt=\"Fig3 Grand Database2019 2019.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/34\/Fig3_Grand_Database2019_2019.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Genealogy tree visualizer interface<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Platform_usage\">Platform usage<\/span><\/h2>\n<p>The LAS platform has been actively employed for research in the Institute for Cancer Research at Candiolo since March 2012. Only a few modules (e.g., BioBanking, Storage) were available in the first release, with a subset of the current features, while the modules addressing the molecular experiments were introduced in 2013. More recently, additional functionalities have been developed to improve the user experience and provide new functionalities to track more information about biological entities and their life cycles. Within our institution, a number of research groups are currently using the platform, and a large amount of data has already been produced and stored. In particular, as of November 2018, researchers have defined approximately 10,800 collections, each one including all biological entities (i.e., aliquots and PDXs) that share a common origin (i.e., the same collection event). The BioBanking module currently stores approximately 300,000 aliquots of different types, as pointed out in Table 1. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 1.<\/b> Aliquots statistics based on their type\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Human\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Xenopatients\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Cell lines\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Complementary DNA<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1414\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">74\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Complementary RNA<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1470\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>DNA<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17,966\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28,240\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">945\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Formalin fixed<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1,454\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22,911\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">336\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Frozen<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">733\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Frozen sediment<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">78\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Labeled section<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">106\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,192\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>OCT frozen<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">338\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Protein<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">129\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Plasma<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">36,220\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">185\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Paraffin section<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">107\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,478\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>RNA Later<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9,455\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">56,493\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>RNA<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">379\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8,449\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">617\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Snap frozen<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1,453\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">49,282\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">134\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Viable<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8,901\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30,126\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10,032\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Total<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>76,867<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>205,578<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>12,278<\/b>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>On average, 3,200 aliquots are generated each month from surgical interventions on human patients, xenopatients, cell line thawing, and derivation procedures. Since the most active user group works with PDXs, ~70% of the aliquots stored are generated from mice. Derived aliquots represent ~20% of the BioBanking module's content, and most of them are DNA. The platform also tracks aliquot consumption according to the types of experiments, as reported in Table 2. Approximately 37% are sent to external laboratories to perform special analyses, while all the other molecular experiments are performed in our institution. The <i>in vivo<\/i> experiments represent one of the main activities of our institution, as well as plasma collection. Indeed, more than 44,000 mice have been tracked. Most of them have already been explanted to generate new aliquots, while approximately 1,700 mice are currently under treatment with experimental drugs, and 1,500 are now implanted to expand the biological material. A detailed categorization of mice statuses is reported in Figure 4. At the moment of writing, more than 15,000 experimental treatments have been applied and 143 are still under execution.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 2.<\/b> Number of aliquots used for each experiment\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Experiment\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\"># of aliquots\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Beaming\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DigitalPCR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Histology-IHC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,939\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Microarray\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">276\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">RealTimePCR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,764\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SangerSequencing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">808\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sequenom\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10,508\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WesternBlots\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">75\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Collaboration (external)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11,264\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Grand_Database2019_2019.png\" class=\"image wiki-link\" data-key=\"330958ff5e42b1921b8f4fb7ef7e2eaf\"><img alt=\"Fig4 Grand Database2019 2019.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2c\/Fig4_Grand_Database2019_2019.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Mice status<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The MDDM module has been widely used in the past years. We tracked more than 15,000 queries submitted by the users from the query generator. This highlights the relevance of this LAS module in the daily activities of our institution.\n<\/p><p>The LAS platform has been changing the research activities in our institution. We periodically ask users about their opinion on diverse system features to understand their impact on the daily work of researchers. Usually, they answer that the system is proving very useful in supporting their research, even if they met some difficulties when they started using the platform. Indeed, they admit that the LAS has changed their working habits (e.g., users of the platform must follow predefined rules and procedures for each operation and instantly report their activity). Furthermore, they noticed an improvement in the data quality and a reduction of the time spent in tracking data, especially when they work in critical environments.\n<\/p><p>Due to the reliability of the system and the coverage of laboratory procedures, the new <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.europdx.eu\/\" target=\"_blank\">EurOPDX Research Infrastructure<\/a> adopted LAS as a LIMS for data management, with LAS to be installed across the six nodes of the distributed infrastructure. Our lab is one of these nodes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The LAS platform is designed to assist researchers of biological and biomedical laboratories in all of their activities. The modular architecture manages heterogeneous and complex data and supports researchers and practitioners in performing different experimental procedures. The graphical interfaces and the web-based architecture are suitable for use in diverse environments, including hostile ones (e.g., in sterile conditions). Due to the modular architecture, a query module has been developed to perform federated queries over all the databases. Thanks to an intuitive graphical interface, the users can design complex queries without any knowledge of databases structures or query languages. In addition, a visualization tool has been integrated in the querying environment to generate targeted visual representations of biological entities and their relationships, allowing the scientists to extract relevant information for further analyses. More visualization tools will be envisioned in the near future.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The EurOPDX Research Infrastructure has received funding from the European Union's Horizon 2020 research and innovation program, grant agreement no. #731105 (EDIReX project). A.B. has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No 724748) and AIRC (IG n 20697 and Special Program 5 per mille metastases Project n 21091).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"System_availability\">System availability<\/span><\/h3>\n<p>LAS can be <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/lasircc\/\" target=\"_blank\">found on GitHub<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>None declared.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-ChenLIMS16-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChenLIMS16_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Chen, Y.; Lin, Y.; Yuan, X. et al.. \"Chapter 9: LIMS and Clinical Data Management\". In Shen, B.; Tang, H.; Jiang, X.. <i>Translational Biomedical Informatics: A Precision Medicine Perspective<\/i>. Springer. pp. 225\u2013240. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-981-10-1503-8_9\" target=\"_blank\">10.1007\/978-981-10-1503-8_9<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9789811015038.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+9%3A+LIMS+and+Clinical+Data+Management&rft.atitle=Translational+Biomedical+Informatics%3A+A+Precision+Medicine+Perspective&rft.aulast=Chen%2C+Y.%3B+Lin%2C+Y.%3B+Yuan%2C+X.+et+al.&rft.au=Chen%2C+Y.%3B+Lin%2C+Y.%3B+Yuan%2C+X.+et+al.&rft.pages=pp.%26nbsp%3B225%E2%80%93240&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-981-10-1503-8_9&rft.isbn=9789811015038&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CLIPaper-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CLIPaper_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wood, S. (September 2007). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20170825181932\/https:\/\/www.it.uu.se\/edu\/course\/homepage\/lims\/vt12\/ComprehensiveLaboratoryInformatics.pdf\" target=\"_blank\">\"Comprehensive Laboratory Informatics: A Multilayer Approach\"<\/a> (PDF). American Laboratory. pp. 3. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.it.uu.se\/edu\/course\/homepage\/lims\/vt12\/ComprehensiveLaboratoryInformatics.pdf\" target=\"_blank\">the original<\/a> on 25 August 2017<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20170825181932\/https:\/\/www.it.uu.se\/edu\/course\/homepage\/lims\/vt12\/ComprehensiveLaboratoryInformatics.pdf\" target=\"_blank\">https:\/\/web.archive.org\/web\/20170825181932\/https:\/\/www.it.uu.se\/edu\/course\/homepage\/lims\/vt12\/ComprehensiveLaboratoryInformatics.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Comprehensive+Laboratory+Informatics%3A+A+Multilayer+Approach&rft.atitle=&rft.aulast=Wood%2C+S.&rft.au=Wood%2C+S.&rft.date=September+2007&rft.pages=pp.+3&rft.pub=American+Laboratory&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20170825181932%2Fhttps%3A%2F%2Fwww.it.uu.se%2Fedu%2Fcourse%2Fhomepage%2Flims%2Fvt12%2FComprehensiveLaboratoryInformatics.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbbottStarlims18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbbottStarlims18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.informatics.abbott\/us\/en\/offerings\/lims\" target=\"_blank\">\"Starlims\"<\/a>. Abbott. 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.informatics.abbott\/us\/en\/offerings\/lims\" target=\"_blank\">https:\/\/www.informatics.abbott\/us\/en\/offerings\/lims<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Starlims&rft.atitle=&rft.date=2018&rft.pub=Abbott&rft_id=https%3A%2F%2Fwww.informatics.abbott%2Fus%2Fen%2Fofferings%2Flims&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SapioSapio18-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SapioSapio18_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sapiosciences.com\/\" target=\"_blank\">\"Sapio Sciences\"<\/a>. Sapio Sciences, LLC. 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sapiosciences.com\/\" target=\"_blank\">https:\/\/www.sapiosciences.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Sapio+Sciences&rft.atitle=&rft.date=2018&rft.pub=Sapio+Sciences%2C+LLC&rft_id=https%3A%2F%2Fwww.sapiosciences.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LabVantageLabVantage18-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LabVantageLabVantage18_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labvantage.com\/\" target=\"_blank\">\"LabVantage\"<\/a>. LabVantage Solutions, Inc. 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.labvantage.com\/\" target=\"_blank\">https:\/\/www.labvantage.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=LabVantage&rft.atitle=&rft.date=2018&rft.pub=LabVantage+Solutions%2C+Inc&rft_id=https%3A%2F%2Fwww.labvantage.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlankenbergGalaxy10-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlankenbergGalaxy10_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blankenberg, D.; Von Kuster, G.; Coraor, N. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4264107\" target=\"_blank\">\"Galaxy: a web-based genome analysis tool for experimentalists\"<\/a>. <i>Current Protocols in Molecular Biology<\/i> <b>19<\/b> (Unit 19.10.1\u201321). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2F0471142727.mb1910s89\" target=\"_blank\">10.1002\/0471142727.mb1910s89<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4264107\/\" target=\"_blank\">PMC4264107<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20069535\" target=\"_blank\">20069535<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4264107\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4264107<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy%3A+a+web-based+genome+analysis+tool+for+experimentalists&rft.jtitle=Current+Protocols+in+Molecular+Biology&rft.aulast=Blankenberg%2C+D.%3B+Von+Kuster%2C+G.%3B+Coraor%2C+N.+et+al.&rft.au=Blankenberg%2C+D.%3B+Von+Kuster%2C+G.%3B+Coraor%2C+N.+et+al.&rft.date=2010&rft.volume=19&rft.issue=Unit+19.10.1%E2%80%9321&rft_id=info:doi\/10.1002%2F0471142727.mb1910s89&rft_id=info:pmc\/PMC4264107&rft_id=info:pmid\/20069535&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4264107&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoecksGalaxy10-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GoecksGalaxy10_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goecks, J.; Nekrutenko, A.; Taylor, J.; The Galaxy Team (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2945788\" target=\"_blank\">\"Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences\"<\/a>. <i>Genome Biology<\/i> <b>11<\/b> (8): R86. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2010-11-8-r86\" target=\"_blank\">10.1186\/gb-2010-11-8-r86<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2945788\/\" target=\"_blank\">PMC2945788<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20738864\" target=\"_blank\">20738864<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2945788\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2945788<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy%3A+a+comprehensive+approach+for+supporting+accessible%2C+reproducible%2C+and+transparent+computational+research+in+the+life+sciences&rft.jtitle=Genome+Biology&rft.aulast=Goecks%2C+J.%3B+Nekrutenko%2C+A.%3B+Taylor%2C+J.%3B+The+Galaxy+Team&rft.au=Goecks%2C+J.%3B+Nekrutenko%2C+A.%3B+Taylor%2C+J.%3B+The+Galaxy+Team&rft.date=2010&rft.volume=11&rft.issue=8&rft.pages=R86&rft_id=info:doi\/10.1186%2Fgb-2010-11-8-r86&rft_id=info:pmc\/PMC2945788&rft_id=info:pmid\/20738864&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2945788&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-O.27ConnorSeqWare10-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-O.27ConnorSeqWare10_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">O'Connor, B.D.; Merriman, B.; Nelson, S.F. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/1471-2105\/11\/S12\/S2\" target=\"_blank\">\"SeqWare Query Engine: storing and searching sequence data in the cloud\"<\/a>. <i>BMC Bioinformatics<\/i> <b>11<\/b> (Suppl. 12): S2. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-11-S12-S2\" target=\"_blank\">10.1186\/1471-2105-11-S12-S2<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.biomedcentral.com\/1471-2105\/11\/S12\/S2\" target=\"_blank\">http:\/\/www.biomedcentral.com\/1471-2105\/11\/S12\/S2<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SeqWare+Query+Engine%3A+storing+and+searching+sequence+data+in+the+cloud&rft.jtitle=BMC+Bioinformatics&rft.aulast=O%27Connor%2C+B.D.%3B+Merriman%2C+B.%3B+Nelson%2C+S.F.&rft.au=O%27Connor%2C+B.D.%3B+Merriman%2C+B.%3B+Nelson%2C+S.F.&rft.date=2010&rft.volume=11&rft.issue=Suppl.+12&rft.pages=S2&rft_id=info:doi\/10.1186%2F1471-2105-11-S12-S2&rft_id=http%3A%2F%2Fwww.biomedcentral.com%2F1471-2105%2F11%2FS12%2FS2&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaralisLAS12-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BaralisLAS12_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Baralis, E.; Bertotti, A.; Fiori, A.; Grand, A. (2012). \"LAS: A software platform to support oncological data management\". <i>Journal of Medical System<\/i> <b>36<\/b> (Suppl. 1): S81\u201390. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10916-012-9891-6\" target=\"_blank\">10.1007\/s10916-012-9891-6<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23117791\" target=\"_blank\">23117791<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LAS%3A+A+software+platform+to+support+oncological+data+management&rft.jtitle=Journal+of+Medical+System&rft.aulast=Baralis%2C+E.%3B+Bertotti%2C+A.%3B+Fiori%2C+A.%3B+Grand%2C+A.&rft.au=Baralis%2C+E.%3B+Bertotti%2C+A.%3B+Fiori%2C+A.%3B+Grand%2C+A.&rft.date=2012&rft.volume=36&rft.issue=Suppl.+1&rft.pages=S81%E2%80%9390&rft_id=info:doi\/10.1007%2Fs10916-012-9891-6&rft_id=info:pmid\/23117791&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ForbesCOSMIC15-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ForbesCOSMIC15_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Forbes, S.A.; Beare, D.; Gunasekaran, P. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383913\" target=\"_blank\">\"COSMIC: Exploring the world's knowledge of somatic mutations in human cancer\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (DB1): D805\u201311. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1075\" target=\"_blank\">10.1093\/nar\/gku1075<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383913\/\" target=\"_blank\">PMC4383913<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25355519\" target=\"_blank\">25355519<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383913\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383913<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=COSMIC%3A+Exploring+the+world%27s+knowledge+of+somatic+mutations+in+human+cancer&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Forbes%2C+S.A.%3B+Beare%2C+D.%3B+Gunasekaran%2C+P.+et+al.&rft.au=Forbes%2C+S.A.%3B+Beare%2C+D.%3B+Gunasekaran%2C+P.+et+al.&rft.date=2015&rft.volume=43&rft.issue=DB1&rft.pages=D805%E2%80%9311&rft_id=info:doi\/10.1093%2Fnar%2Fgku1075&rft_id=info:pmc\/PMC4383913&rft_id=info:pmid\/25355519&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383913&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ForbesAbstract16-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ForbesAbstract16_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Forbes, S.A.; Bindal, N.; Beare, D. et al. (2016). \"Abstract 5285: COSMIC: Comprehensively exploring oncogenomics\". <i>Cancer Research<\/i> <b>76<\/b> (14 Suppl.): 5285. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1158%2F1538-7445.AM2016-5285\" target=\"_blank\">10.1158\/1538-7445.AM2016-5285<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Abstract+5285%3A+COSMIC%3A+Comprehensively+exploring+oncogenomics&rft.jtitle=Cancer+Research&rft.aulast=Forbes%2C+S.A.%3B+Bindal%2C+N.%3B+Beare%2C+D.+et+al.&rft.au=Forbes%2C+S.A.%3B+Bindal%2C+N.%3B+Beare%2C+D.+et+al.&rft.date=2016&rft.volume=76&rft.issue=14+Suppl.&rft.pages=5285&rft_id=info:doi\/10.1158%2F1538-7445.AM2016-5285&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KittsTheData13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KittsTheData13_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Kitts, A.; Phan, L.; Ward, M. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK174586\/\" target=\"_blank\">\"The Database of Short Genetic Variation (dbSNP)\"<\/a>. <i>The NCBI Handbook<\/i> (2nd ed.). National Center for Biotechnology Information<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK174586\/\" target=\"_blank\">https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK174586\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Database+of+Short+Genetic+Variation+%28dbSNP%29&rft.atitle=The+NCBI+Handbook&rft.aulast=Kitts%2C+A.%3B+Phan%2C+L.%3B+Ward%2C+M.+et+al.&rft.au=Kitts%2C+A.%3B+Phan%2C+L.%3B+Ward%2C+M.+et+al.&rft.date=2013&rft.edition=2nd&rft.pub=National+Center+for+Biotechnology+Information&rft_id=https%3A%2F%2Fwww.ncbi.nlm.nih.gov%2Fbooks%2FNBK174586%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EilbeckTheSeq05-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EilbeckTheSeq05_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Eilbeck, K.; Lewis, S.E.; Mungall, C.J. et al. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1175956\" target=\"_blank\">\"The Sequence Ontology: A tool for the unification of genome annotations\"<\/a>. <i>Genome Biology<\/i> <b>6<\/b> (5): R44. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2005-6-5-r44\" target=\"_blank\">10.1186\/gb-2005-6-5-r44<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1175956\/\" target=\"_blank\">PMC1175956<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15892872\" target=\"_blank\">15892872<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1175956\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1175956<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Sequence+Ontology%3A+A+tool+for+the+unification+of+genome+annotations&rft.jtitle=Genome+Biology&rft.aulast=Eilbeck%2C+K.%3B+Lewis%2C+S.E.%3B+Mungall%2C+C.J.+et+al.&rft.au=Eilbeck%2C+K.%3B+Lewis%2C+S.E.%3B+Mungall%2C+C.J.+et+al.&rft.date=2005&rft.volume=6&rft.issue=5&rft.pages=R44&rft_id=info:doi\/10.1186%2Fgb-2005-6-5-r44&rft_id=info:pmc\/PMC1175956&rft_id=info:pmid\/15892872&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1175956&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VukoticNeo4j14-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VukoticNeo4j14_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Vukotic, A.; Watt, N.; Abedrabbo, T. et al. (2014). <i>Neo4j in Action<\/i>. Manning Publications. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781617290763.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Neo4j+in+Action&rft.aulast=Vukotic%2C+A.%3B+Watt%2C+N.%3B+Abedrabbo%2C+T.+et+al.&rft.au=Vukotic%2C+A.%3B+Watt%2C+N.%3B+Abedrabbo%2C+T.+et+al.&rft.date=2014&rft.pub=Manning+Publications&rft.isbn=9781617290763&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BertottiAMolec11-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BertottiAMolec11_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bertotti, A.; Migliardi, G.; Galimi, F. et al. (2011). \"A molecularly annotated platform of patient-derived xenografts (\"xenopatients\") identifies HER2 as an effective therapeutic target in cetuximab-resistant colorectal cancer\". <i>Cancer Discovery<\/i> <b>1<\/b> (6): 508\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1158%2F2159-8290.CD-11-0109\" target=\"_blank\">10.1158\/2159-8290.CD-11-0109<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22586653\" target=\"_blank\">22586653<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+molecularly+annotated+platform+of+patient-derived+xenografts+%28%22xenopatients%22%29+identifies+HER2+as+an+effective+therapeutic+target+in+cetuximab-resistant+colorectal+cancer&rft.jtitle=Cancer+Discovery&rft.aulast=Bertotti%2C+A.%3B+Migliardi%2C+G.%3B+Galimi%2C+F.+et+al.&rft.au=Bertotti%2C+A.%3B+Migliardi%2C+G.%3B+Galimi%2C+F.+et+al.&rft.date=2011&rft.volume=1&rft.issue=6&rft.pages=508%E2%80%9323&rft_id=info:doi\/10.1158%2F2159-8290.CD-11-0109&rft_id=info:pmid\/22586653&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShethFed90-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShethFed90_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sheth, A.P.; Larson, J.A. (1990). \"Federated database systems for managing distributed, heterogeneous, and autonomous databases\". <i>ACM Computing Surveys<\/i> <b>22<\/b> (3): 183\u2013236. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F96602.96604\" target=\"_blank\">10.1145\/96602.96604<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Federated+database+systems+for+managing+distributed%2C+heterogeneous%2C+and+autonomous+databases&rft.jtitle=ACM+Computing+Surveys&rft.aulast=Sheth%2C+A.P.%3B+Larson%2C+J.A.&rft.au=Sheth%2C+A.P.%3B+Larson%2C+J.A.&rft.date=1990&rft.volume=22&rft.issue=3&rft.pages=183%E2%80%93236&rft_id=info:doi\/10.1145%2F96602.96604&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AtzeniData99-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AtzeniData99_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Atzeni, P.; Ceri, S.; Paraboschi, S.; Torlone, R. (1999). <i>Database Systems: Concepts, Languages & Architectures<\/i>. The McGraw-Hill Companies. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 0077095006.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Database+Systems%3A+Concepts%2C+Languages+%26+Architectures&rft.aulast=Atzeni%2C+P.%3B+Ceri%2C+S.%3B+Paraboschi%2C+S.%3B+Torlone%2C+R.&rft.au=Atzeni%2C+P.%3B+Ceri%2C+S.%3B+Paraboschi%2C+S.%3B+Torlone%2C+R.&rft.date=1999&rft.pub=The+McGraw-Hill+Companies&rft.isbn=0077095006&rfr_id=info:sid\/en.wikipedia.org:Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204343\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.514 seconds\nReal time usage: 2.155 seconds\nPreprocessor visited node count: 14153\/1000000\nPreprocessor generated node count: 34171\/1000000\nPost\u2010expand include size: 106690\/2097152 bytes\nTemplate argument size: 35637\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 301.366 1 - -total\n 81.23% 244.793 1 - Template:Reflist\n 67.53% 203.515 17 - Template:Citation\/core\n 39.50% 119.027 9 - Template:Cite_journal\n 19.30% 58.171 4 - Template:Cite_book\n 13.14% 39.605 4 - Template:Cite_web\n 12.66% 38.146 1 - Template:Infobox_journal_article\n 12.06% 36.331 1 - Template:Infobox\n 8.18% 24.642 80 - Template:Infobox\/row\n 7.82% 23.566 23 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11392-0!*!0!!en!5!* and timestamp 20200707204341 and revision id 37575\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform\">https:\/\/www.limswiki.org\/index.php\/Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","45b5e684bad1c44cd9a1cb43bb6b311e_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/bd\/Fig1_Grand_Database2019_2019.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a3\/Fig2_Grand_Database2019_2019.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/34\/Fig3_Grand_Database2019_2019.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2c\/Fig4_Grand_Database2019_2019.png"],"45b5e684bad1c44cd9a1cb43bb6b311e_timestamp":1594154621,"6fa400ac6b9e1784eed63829a3731260_type":"article","6fa400ac6b9e1784eed63829a3731260_title":"Information technology and medical technology personnel\u2019s perception regarding segmentation of medical devices: A focus group study (Johansson et al. 2020)","6fa400ac6b9e1784eed63829a3731260_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study","6fa400ac6b9e1784eed63829a3731260_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Information technology and medical technology personnel\u2019s perception regarding segmentation of medical devices: A focus group study\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nInformation technology and medical technology personnel\u2019s perception regarding segmentation of medical devices: A focus group studyJournal\n \nHealthcareAuthor(s)\n \nJohansson, David; J\u00f6nsson, Patrik; Ivarsson, Bodil; Christiansson, MariaAuthor affiliation(s)\n \nEdith Cowan University, Embry-Riddle Aeronautical UniversityPrimary contact\n \nEmail: david dot a dot johansson at skane dot seYear published\n \n2020Volume and issue\n \n8(1)Page(s)\n \n23DOI\n \n10.3390\/healthcare8010023ISSN\n \n2227-9032Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/2227-9032\/8\/1\/23Download\n \nhttps:\/\/www.mdpi.com\/2227-9032\/8\/1\/23\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Materials and methods \n\n3.1 Design and setting \n3.2 Informants \n3.3 Ethics \n3.4 Data collection \n3.5 Data analysis \n\n\n4 Results \n\n4.1 Information security \n\n4.1.1 Outsider threats \n4.1.2 Insider threats \n\n\n4.2 Implementation of segmentation in healthcare \n\n4.2.1 Predictions of an increase in costs and administration \n4.2.2 Predictions of required knowledge improvement for the healthcare organization \n4.2.3 Possibilities and pitfalls concerning segmentation \n4.2.4 Increased demands on medical device manufacturers \n\n\n\n\n5 Discussion \n6 Conclusion \n7 Acknowledgements \n\n7.1 Author contributions \n7.2 Funding \n7.3 Conflict of interest \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nObjective: Segmentation is one way of improving data protection. The aim of this study was to investigate information technology (IT) and medical technology (MT) personnel\u2019s perception in relation to ongoing segmentation of medical devices and IT infrastructure in the healthcare sector. \nMethods: Focus group interviews with nine IT and nine MT personnel in a county council in southern Sweden were conducted. The interviews focused on two areas: positive expectations and misgivings. Digital recordings were transcribed verbatim and analyzed using qualitative content analysis. \nResults: Responses related to two main areas: information security and implementation of segmentation. Informants stated that network segmentation would increase the overall level of cybersecurity for medical devices, addressing both insider and outsider threats. However, it would also increase the need for administration and the need for knowledge of the communication patterns of medical devices from the manufacturer\u2019s perspective. \nConclusion: IT and MT personnel in a county council in southern Sweden believed that segmentation would increase cybersecurity but also increase administration and resource needs, which are important opinions to take into consideration. The present study can be used as a model for others to increase awareness of opinions of healthcare organizations. \nKeywords: cybersecurity, healthcare technology, patient safety, staff attitudes\n\nIntroduction \nMedical technology (MT) devices\u2014as defined by the European Union's directive on medical devices[1] and information technology (IT) and device developers\u2014serve an increasingly central role in clinical practice, improving patient health, safety, and quality of life. And the number of such medical devices that are connected to a network continues to grow.[2][3][4]\nNetworked medical devices can also cause substantial harm since they have not historically been designed with a focus on cybersecurity.[5] Errors that underlie device-related injuries are often categorized into three types: manufacturer-related errors, use or design errors, and user errors.[6] It has been said that the weakest link in the process is the user, who must understand how to configure and use medical technology products correctly to achieve a high level of security in computing infrastructure.[7][8]\nCybersecurity is today one of the most important security-related challenges for all countries, but its visibility and awareness are still limited to the public, although almost everyone has heard something about it.[9] There is a substantial security risk posed by outsiders identifying potential interactions between the interconnected elements in hospital systems and computing infrastructures, and then taking advantage of poor cybersecurity to steal medical health records, deny access to health services, or cause intentional harm with these.[10][11] For medical devices specifically, it may be possible for hackers to use them in order to gain access to confidential patient data and to reprogram them to send harmful commands.[12] This could mean, for example, altering blood groups or test results, or taking control of pumps that regulate the administration of potent drugs.[13] Other harmful activities that could be enacted include disrupting the parameter values used in the scanning protocols, tampering with the radiation exposure levels, causing mechanical disruption, and creating denial-of-service attacks.[14]\nOther researchers have also investigated the subject of cybersecurity in medical devices and have concluded that there is a relationship between the increase in network connectable medical devices and increased cybersecurity risks.[15] As such, segmenting the network into multiple layers, with security gates such as firewalls in between them, could be an effective way to contain network problems and reduce the impact of a breach in network security.[16][17] This is comparable to dividing a building into fire zones to delay fire spreading, and enabling firefighting in sections rather than in the building as a whole.[18]\nApplying the concept of network segmentation to medical devices may reduce the associated risks and prevent intrusion. This entails separating elements such as computers, servers, routers, data, and healthcare personnel into groups, thereby restricting access and better protecting vital services.[19][20] Segmentation of medical devices is described as a good method for ensuring data security and is recommended to be used to whatever extent is feasible.[21] In order to increase understanding of innovations in healthcare organizations, such as segmentation of medical devices, L\u00e4nsisalmi et al.[22] argue that health innovations should be investigated from the perspectives of stakeholders, in this case IT and MT personnel. Therefore, the aim of this study was to investigate IT and MT personnel\u2019s perception of segmentation of medical devices and IT infrastructure in the healthcare sector.\nHighlights of this study:\n\n Medical devices can cause substantial harm. \n The user of medical devices must understand how to achieve a high level of security. \n Hackers might be able to reprogram medical devices to send harmful commands. \n One way to reduce the risk of hackers and prevent intrusion is network segmentation. \n MT and IT personnel must be positive about and open to the improvement of cybersecurity.\nMaterials and methods \nDesign and setting \nA qualitative methodology design was employed with focus group sessions in order to gain a deeper insight into the views and experiences of IT segmentation. This descriptive study was conducted at Region Sk\u00e5ne, one of the 21 county councils\/regions in Sweden. Region Sk\u00e5ne employs around 34,000 people, mostly in healthcare, and has the main responsibility for public healthcare and medical services in the region.[23] Much of the healthcare is conducted in a high-tech environment, which includes devices such as patient monitoring equipment, medical imaging systems, and laboratory instrumentation. A number of cybersecurity-related incidents prompted a decision to implement network segmentation for all devices in Region Sk\u00e5ne. The study was performed during the ongoing implementation of the network segmentation of medical devices.\n\nInformants \nInformants were strategically recruited from IT and MT personnel based on their experience working with medical devices such as patient monitoring equipment, medical imaging systems, and laboratory instrumentation in Region Sk\u00e5ne to ensure maximal variation in socio-demographic data and service positions.[24] The following variables were taken into account to ensure a broad selection: age, time in the profession, and competence in (1) segmentation, (2) MT device systems, (3) firewall-based traffic filtering, and (4) IT security.\n\nEthics \nThe study conformed to the principles outlined in the Declaration of Helsinki, and according to Swedish legislation (SFS 2003:460) no ethical approval was required as the study did not involve patients. The project was approved by the directors of the participating departments. All informants were informed of the aims and procedures of the study through a short written presentation included in an email invitation. Informed consent was considered implied when informants signed up voluntarily for the focus group interviews. The data processing was carried out in accordance to the General Data Protection Regulation (GDPR).\n\nData collection \nThree mixed MT and IT focus group interviews[25] were held with five to eight informants in each group and included a total of 18 informants (nine MT personnel and nine IT personnel; see Table 1 for demographic and other characteristics). Interviews were conducted in a conference room and took about 90 minutes each. The discussions were digitally recorded (with the informants\u2019 consent) and independently transcribed by a trusted agency.\n\n\n\n\n\n\n\nTable 1. Demographic data and other characteristics\n\n\nGender\n\n# of informants\n\n\nWomen\n\n3\n\n\nMen\n\n15\n\n\nStatistical measure of age\n\nResult\n\n\nMean standard deviation\n\n52 \u00b1 10\n\n\nMedian (range)\n\n51 (33\u201367)\n\n\nStatistical measure of years\r\nof professional experience\n\nResult\n\n\nMean standard deviation\n\n18 \u00b1 12\n\n\nMedian (range)\n\n17 (2\u201340)\n\n\nHighest level of education\n\n# of informants\n\n\nHigh school (n)\n\n3\n\n\nUniversity (n)\n\n15\n\n\n\nThe interviews were conducted with one researcher who had extensive experience in focus group interviews acting as moderator and a second acting as assistant. A short presentation of network segmentation was held prior to discussions as an introduction to the topic. A rough outline of network segmentation and specifically how it relates to the segmentation of medical devices was also presented (see Figure 1). The interviews focused on positive expectations and misgivings about the solution. Informants were asked two open-ended questions: \u201cWhat are the advantages of segmentation of medical devices?\u201d and \u201cWhat misgivings do you have regarding segmentation of medical devices?\u201d Probing questions were also used.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Example of rough outline of network segmentation\n\n\n\nData analysis \nThe text from the interviews was analyzed using qualitative content analysis.[26][27] Transcribed interviews were first read as a whole by all authors to gain an overall understanding. The text was then divided into meaningful units, which were condensed, coded, and gathered into groups with similar content. These steps were carried out by the authors individually, and then all authors gathered and discussed the findings. These were then abstracted into subcategories that met the study aim. Finally, the subcategories were abstracted into two categories. An example of the analysis process can be found in Table 2. The statements were thoroughly assessed by the authors to ensure that no statements relating to positive expectations or misgivings regarding segmentation were left out due to the condensation of meaning units into codes.\n\r\n\n\n\n\n\n\n\n\nTable 2. Example of the analysis process, from verbatim meaning unit to category\n\n\nMeaning unit\n\nCondensed\n\nCode\n\nSubcategory\n\nCategory\n\n\n\u2026So theoretically, if somebody would have a bad intent, it\u2019s very easy today. Considering that today the risk is greater internally than externally. And it doesn\u2019t have to be malicious, it may be that you make a mistake because you have access to things that you should not have access to. And then that\u2019s really a good thing then, to segment, we clearly get what\u2019s stealing from and we can follow it up in a new way\u2026 (Participant 2)\n\n\u201c\u2026it doesn\u2019t have to be malicious, of course, you might make a mistake just because you have access to things that you shouldn\u2019t have\u201d (Participant 2)\n\nImproving patient security\n\nInsider threats\n\nInformation security\n\n\n\u201cI think it will require a great deal of resources to get it implemented. I see it as very resource-intensive to implement\u2026 longer lead times and complicated administration. There is, as we stated, not enough resources to handle requests in firewall changes\u2026\u201d (Participant 17)\n\n\u201cI see it [implementation] as very resourceintensive to implement \u2026 longer lead times and complicated administration\u201d (Participant 17)\n\nResource-intensive project\n\nPredictions of an increase in costs and administration\n\nImplementation of segmentation in healthcare\n\n\n\nResults \nTwo categories emerged during the analysis of the transcribed interviews: \u201cInformation security\u201d and \u201cImplementation of segmentation in healthcare.\u201d These categories contained six subcategories (Table 3).\n\n\n\n\n\n\n\nTable 3. Distribution of categories and subcategories concerning information technology (IT) and medical technology (MT) personnel views in relation to segmentation of medical devices and IT infrastructure in the healthcare sector (N = 18).\n\n\nCategories\n\nSubcategories\n\n\nInformation security\n\nOutsider threats\n\n\nImplementation of segmentation in healthcare\n\n\nInsider threats\r\n\nPredictions of an increase in costs and administration\r\n\nPredictions of required knowledge improvement for the healthcare organization\r\n\nPossibilities and pitfalls concerning segmentation\n\n\n\n\nInformation security \nOutsider threats \nAcross the three focus groups, attacks originating from outside of the healthcare organization were well known, and the informants had personal experience of such attacks or had heard of them from others. Virus attacks that led to numerous problems and increased work had been experienced. Informants were, on the whole, positive to the possibilities of limiting data virus infections that are offered by segmentation technology, as well as the potential increase in patient safety. They were also positive to the increased protection that segmentation offered to older applications unable to withstand outsider threats due to patching policies or lack of security design.\nIt was stated that segmentation reduces the risk that core systems or patients will be affected during an infection of malicious software or hacker attacks\/intrusions, primarily by restricting access between technical devices.\nAs Informant 3 put it: \u201cThe thing is, that you can divide up and protect \u2026 When you look at virus attacks and other things, that it doesn\u2019t affect different segments but stays within the same segment \u2026 That\u2019s the advantage as I see it.\u201d\n\nInsider threats \nIn relation to insider threats, informants had experienced personnel accidentally forwarding infected emails, which caused great harm. They perceived that segmentation would reduce the risk of users getting hold of restricted information via administrative interfaces that they should not have access to, thereby reducing the probability of them making administrative mistakes. It also would reduce the probability that users, whether by mistake or intention, introduced malicious code into medical devices and\/or core infrastructure. The informants opined that with segmentation, devices would be less likely to be incorrectly configured when connected to the network, thereby reducing the likelihood of failures or connection issues and, in turn, improving patient security. Informant 2 adds: \u201c\u2026 It doesn\u2019t have to be malicious, of course, you might make a mistake just because you have access to things that you shouldn\u2019t have.\u201d\nHowever, the informants expressed the benefits of segmentation would be cancelled out, however, due to the risk that so many openings would be needed between segments due to the large number of older devices.\n\nImplementation of segmentation in healthcare \nPredictions of an increase in costs and administration \nThe informants, in general, predicted an increase in administration and costs for the organization. Because of the higher level of complexity introduced by segmentation, the informants predicted that working with a segmented network would require better planning and documentation. A positive side effect of this would be a better understanding of future costs for the healthcare organization, which would be especially useful for detecting increased costs due to deviating or nonstandard network communication solutions for medical technology. It was also expected that segmentation would lead to better planned maintenance work for medical technology solutions since network communication would be more controlled and structured.\nThe informants assumed that implementing segmentation would increase administrative tasks and the need for resources (i.e., make it time consuming) due to the increased number of firewalls between segments, and openings needed in these. Also, new processes, routines, and administrative tools would be needed, such as automated software. There were, therefore, misgivings that segmentation would lead to longer execution times and that increased planning would be required for network changes.\nInformant 17 puts it as such: \u201cI see it [implementation] as very resource-intensive to implement \u2026 [with] longer lead times and complicated administration.\u201d\n\nPredictions of required knowledge improvement for the healthcare organization \nThe informants perceived that their own knowledge and competence regarding network communication would need to improve, due to a higher level of complexity. Informants had positive expectations of receiving the necessary education and assumed that management felt the same way. However, the informants also stated the importance of informing all relevant parties about the strategy and goal of the network segmentation, including, for example, personnel affiliated with system management and medical technology devices within the healthcare organization.\nInformant 1 states: \u201cI see an advantage and a disadvantage in the fact that more knowledge is required \u2026 not only from us, but also from our suppliers \u2026 With segmentation, we will need a better understanding of all of our systems.\u201d\n\nPossibilities and pitfalls concerning segmentation \nThe informants perceived that the segmentation solution provided an opportunity to use the infrastructure to provide restricted communication rights based on actual needs of users and systems, which would provide a high degree of flexibility and thereby better preparedness for new communication requirements. It also presented an easier way to reuse the infrastructure. They also expressed that the design gave an advantage by isolating attacks through the network in an easier way, and that it gave the basis for a more controlled and secure network operations environment by reducing the impact of changes and faults between network segments. Furthermore, they said that dedicated network segments for remote administration would give fewer attack vectors between systems.\nThe informants highlighted the need to communicate overall policies and designs, as well as guidelines for implementation of segmentation, to avoid counterproductive effects. One such effect could be creating too many openings between the segments so that the key benefits of segmentation would be lost.\nPer Informant 18: \u201cSegmentation is really about trying to minimize consequences \u2026 If we discover that one segment has a virus \u2026 then we shut down the framework, which means that it\u2019s completely isolated.\u201d\n\nIncreased demands on medical device manufacturers \nThe informants had also experienced working with medical device manufacturers who connect to medical devices remotely. It was foreseen that requirements for knowledge, planning ability, and documentation from manufacturers of how their devices work and how they integrate with other devices would increase once segmentation was completed. This would give increased insight into devices, something that informants perceived as currently lacking, both in customers and manufacturers. However, this could also lead to increased cost. \u201c\u2026 And then I think another problem could be that our suppliers, they don\u2019t always know exactly how their applications talk to each other, which ports are used. So that could also be challenging to get right,\u201d adds Informant 15.\nSegmentation was also expected to give an opportunity for manufacturers to connect to devices for support in more diverse and secure ways. For instance, it would be possible to allow forms of communication other than virtual private network (VPN) without affecting the overall security level. It would also be possible to reduce manufacturer\u2019s access rights to other systems, thus reducing the risk of manufacturer-induced faults (paraphrasing of comments from Informant 5).\n\nDiscussion \nThis qualitative study provides insights into the perceptions among IT and MT personnel after a decision to implement network segmentation of medical devices and IT infrastructure in the healthcare sector. Despite the fact that this is a small study, the results corroborated that healthcare systems are complex structures, and medical devices that are integrated into a hospital\u2019s IT systems are just a minor part of the system as a whole. However, these devices play a critical role in clinical function. If a medical device fails, there are many potential hazards to patient safety. There are many techniques for increasing cybersecurity, including not only network segmentation but also security policies, antivirus software, lifecycle management of operating systems, security patches, drivers, and so on.[15] Hopefully, the present study may partly provide support and inspire healthcare organizations to develop and communicate a strategy that encompasses all these important parts. Generally, security can be further enhanced if the segmentation is implemented with rigorous and strictly enforced security controls regarding communication between segments. As such, the communication boundaries between segments will not stop cyberattacks entirely unless the network also complies with rules, standards, and requirements that emphasize access control policies using a so called \u201cnext-generation firewall\u201d (NGFW) implementation, which includes additional network device-filtering functionalities such as an intrusion prevention system (IPS), intrusion detection protocol (IDP), activity logging, and web and mail security features. A simple application layer firewall cannot stop camouflaged malware from causing security risks to healthcare facilities.\nIn this study, which focused specifically on expectations and misgivings regarding segmentation, there were two themes that emerged across the categories identified in the analysis, namely information security and implementation of segmentation in healthcare. The informants in the study had experience of insider and outsider threats. In May 2017, a major ransomware attack called WannaCry occurred, which affected 200,000 systems in around 150 countries worldwide.[13] In the U.K., 50 hospitals were affected, patient safety was threatened, and there were calls to highlight cybersecurity in healthcare.[13] In the present study, informants assumed that insider and outsider threats and threats from malicious software, such as ransomware, would decrease with the use of segmentation. Besides the advantages pointed out by the informants, segmentation adds additional security measures to hinder attack vectors. An example is the possibility to deny certain types of network traffic on layer seven of the Open Systems Interconnection (OSI)-stack (i.e., message handling in the application layer), which allows detection of communication patterns. This will decrease the spread of malicious code and denial-of-services threats. Medical devices are regarded to have a low capability of defending themselves against these sorts of threats and will benefit greatly from the added security layers. It could be speculated that if all England's hospitals had implemented segmentation of medical equipment ahead of the WannaCry attack[13], the impact would probably have been lessened as the dispersion could have been limited. If this were the case, hospitals could have continued to produce care and thus maintained a higher level of patient safety, especially if there had been support for technologies such as an intrusion prevention system (IPS). However, healthcare organizations need to be aware that segmentation, although important, is just one measure to increase cybersecurity; relying on it solely could give a false sense of security.\nInformants in the study opined that with segmentation, the demand for documentation would increase regarding communication between infrastructure components within and between systems. It was also opined that the number of firewall changes (i.e., allowed network traffic patterns between segments) and the overall workload of the network and system administrators would increase, which, in turn, would lead to technical skills and increased costs. These opinions are important to take into account from a change management perspective as the informants represent a critical success factor in achieving a high cybersecurity level. Under-investing in cybersecurity is not unique for Sweden. It has previously been pointed out that compared to other sectors that spend four to ten percent of their budget, the healthcare sector spends a relatively small amount (one to two percent) of its budget on IT infrastructure. This must be reconsidered so that fundamental patient safety, healthcare technology, and functionality are not compromised.[13][28] It can be stated that, as healthcare organizations are relying more and more on digital technologies, both regarding medical and other systems, the need for cybersecurity will also become more aligned with other sectors, such as industries and banks.\nOn the basis of the present study, the suggestion of a more structured way of working can be perceived as less flexible with regard to changes, from an administrative point of view, while requiring new processes and new supporting software. On the positive side, segmentation will give better insight into the communication landscape, and a better overview. Historically, cyberattacks have had a significant impact on production, generating high costs for the targeted organization; the healthcare sector must find cost-effective ways to prevent such potentially devastating costs.[13] It must also be assumed that the confidence of stakeholders (patients, manufacturers, and institutions) in the healthcare organization will be reduced after a successful cyberattack.\nOur results showed a need for improvement in the level of competence, as the informants assumed that a greater knowledge base would be required due to the higher level of complexity associated with segmentation. This corroborates a Swedish study showing that a majority of IT and MT personnel considered recurrent training on IT and MT safety, rules, and regulations, as well as a risk analysis with a focus on patient safety, to be of great importance.[7] It is also important that the goals and importance of segmentation are communicated throughout the organization, and the processes and guidelines established regarding system placement within the different segments. The healthcare organization must commit to and supply the means for education and training.\nThe informants in this study assumed that segmentation would provide more efficient and flexible ways of working for healthcare personnel since changing network sockets will not result in additional administrative tasks. Also, dedicated segments for remote administration add better control and logging capabilities in relation to external users, such as medical device suppliers providing support. The informants also identified a great need for thorough planning before the implementation of segmentation, to make the administration, configuration, and traffic analysis manageable for the organization. The number of segments increases the amount of hardware required, and it is also important to plan where medical devices are placed to minimize administration between segments.\nIn order to meet the needs of cybersecurity of the future, the present study stated that the knowledge level of medical device manufacturers, in general, needed to be increased in association with segmentation. It has earlier been highlighted that increased collaboration between medical device manufacturers and personnel in the healthcare sector is fundamental to ensure effective protection.[29] One effect of segmentation is that suppliers will gain new possibilities to connect to their medical devices within the healthcare organization, for example, during support cases. Segmentation will not reduce the regulation and security requirements that suppliers need to commit to. This is in agreement with[30] that stated the manufactures had challenges to meet regulation and safety standards including lifecycle management to improve medical device cybersecurity. It is, therefore, important that requirements regarding connection for remote maintenance and support are clearly agreed to during the acquisition process for new medical devices.\nWe believe mixed IT\/MT focus groups increased coverage of the subject area and provided openness in discussions regarding to being better able to extract novel insights and various expert knowledge. However, because of the study\u2019s qualitative nature, the aim was not to generalize the results to the healthcare sector as a whole. However, it is reasonable to assume that the results show a part of the reality of healthcare in one county council as well as give a better understanding of cybersecurity issues. A review on cybersecurity stated that there are very few studies in the area which include human and organizational aspects, as well as strategy and management. This may justify the present study.[31] A question we have asked ourselves is whether the outcome would have been different if the interviews had been held in the near future after an attack like the WannaCry ransomware attack.[13] That attack is one more indication that cybersecurity policies must be in place for proactive use in the healthcare section and widely communicated, even if the cost seems high. Further, a large (country-scale) study of the implementation of network segmentation would be of interest for further research, and also a study of the health economic impact of a cyberattack.\n\nConclusion \nNetwork segmentation decreases the probability of spreading malicious software and intrusions through a network by introducing barriers between network areas, much in the same way fire zones decreases the probability for fire spreading in a building. Medical devices have historically not had a focus on security features. In this study, it was apparent that MT and IT personnel were positively receptive to the increase in cybersecurity provided by network segmentation but concerned about the increase in the administration that it will entail for medical devices. These opinions are important to take into account to be able to reach the desired increase in cybersecurity. Cybersecurity risks are multifaceted. The present study can be used as a model for other clinical healthcare manufacturers to increase awareness of concerns and opinions in personnel and healthcare organizations in general.\n\nAcknowledgements \nWe are grateful to the informants who took part in this research and would also like to thank the Medical Services, Region Sk\u00e5ne, Sweden, for their support.\n\nAuthor contributions \nConceptualization, B.I. and D.J.; methodology, B.I. and M.C.; formal analysis, D.J., B.I., P.J. and M.C.; investigation, B.I. and M.C.; writing\u2014original draft preparation, B.I., D.J. and M.C.; writing\u2014 review and editing, D.J., B.I., P.J. and M.C. All authors have read and agreed to the published version of the manuscript.\n\nFunding \nThis research received no external funding.\n\nConflict of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 European Commission (14 June 1993). \"Council Directive 93\/42\/EEC of 14 June 1993 concerning medical devices\". EUR-Lex. https:\/\/eur-lex.europa.eu\/eli\/dir\/1993\/42\/2007-10-11 . Retrieved 20 January 2020 .   \n\n\u2191 Abernethy, A.P.; Wheeler, J.L.; Bull, J. (2011). \"Development of a health information technology-based data system in community-based hospice and palliative care\". American Journal of Preventative Medicine 40 (5 Suppl. 2): S217\u201324. doi:10.1016\/j.amepre.2011.01.012. PMID 21521597.   \n\n\u2191 McCaffery, F.; Burton, J.; Richardson, I. (2010). \"Risk management capability model for the development of medical device software\". Software Quality Journal 18: 18. doi:10.1007\/s11219-009-9086-7.   \n\n\u2191 Tulasidas, S.; Mackay, R.; Craw, P. et al. (2013). \"Process of Designing Robust, Dependable, Safe and Secure Software for Medical Devices: Point of Care Testing Device as a Case Study\". Journal of Software Engineering and Applications 6 (9A): 1\u201313. doi:10.4236\/jsea.2013.69A001.   \n\n\u2191 Arney, D.; Venkatasubramanian, K.K.; Sokolsky, O.; Lee, I. (2011). \"Biomedical devices and systems security\". Conference Proceedings of the IEEE Engineering in Medicine and Biology Society 2011: 2376-9. doi:10.1109\/IEMBS.2011.6090663. PMID 22254819.   \n\n\u2191 Jha, A.K.; Prasopa-Plaizier, N.; Larizgoitia, I. et al. (2010). \"Patient safety research: an overview of the global evidence\". Quality and Safety in Health Care 19 (1): 42\u20137. doi:10.1136\/qshc.2008.029165. PMID 20172882.   \n\n\u2191 7.0 7.1 Ivarsson, B.; Wiinberg, S.; Svensson, M. et al. (2016). \"Information and\/or medical technology staff experience with regulations for medical information systems and medical devices\". Health Policy and Technology 5 (4): 383\u201388. doi:10.1016\/j.hlpt.2016.07.008.   \n\n\u2191 Nurse, J.R.C.; Buckley, O.; Legg, P.A. et al. (2014). \"Understanding Insider Threat: A Framework for Characterising Attacks\". 2014 IEEE Security and Privacy Workshops: 214-228. doi:10.1109\/SPW.2014.38.   \n\n\u2191 de Brujin, H.; Janssen, M. (2017). \"Building Cybersecurity Awareness: The need for evidence-based framing strategies\". Government Information Quarterly 34 (1): 1\u20137. doi:10.1016\/j.giq.2017.02.007.   \n\n\u2191 Fischbacher-Smith, D.; Fischbacher-Smith, M. (2013). \"The Vulnerability of Public Spaces: Challenges for UK hospitals under the \u2018new\u2019 terrorist threat\". Public Management Review 15 (3): 330\u201343. doi:10.1080\/14719037.2013.769851.   \n\n\u2191 Coventry, L.; Branley, D. et al. (2018). \"Cybersecurity in healthcare: A narrative review of trends, threats and ways forward\". Maturitas 113: 48\u201352. doi:10.1016\/j.maturitas.2018.04.008. PMID 29903648.   \n\n\u2191 Leavitt, N. (2010). \"Researchers Fight to Keep Implanted Medical Devices Safe from Hackers\". Computer 43 (8): 11\u201314. doi:10.1109\/MC.2010.237.   \n\n\u2191 13.0 13.1 13.2 13.3 13.4 13.5 13.6 Martin, G.; Martin, P.; Hankin, C. et al. (2017). \"Cybersecurity and healthcare: How safe are we?\". BMJ 358: j3179. doi:10.1136\/bmj.j3179.   \n\n\u2191 Mahler, T.; Nissim, N.; Shalom, E. et al.. \"Know Your Enemy: Characteristics of Cyber-Attacks on Medical Imaging Devices\". arXiv. https:\/\/arxiv.org\/abs\/1801.05583 .   \n\n\u2191 15.0 15.1 Schwartz, S.; Ross, A.; Carmody, S. et al. (2018). \"The Evolving State of Medical Device Cybersecurity\". Biomedical Instrumentation and Technology 52 (2): 103-111. doi:10.2345\/0899-8205-52.2.103. PMID 29558182.   \n\n\u2191 Reichenberg, N. (20 March 2014). \"Improving Security via Proper Network Segmentation\". Security Week. https:\/\/www.securityweek.com\/improving-security-proper-network-segmentation . Retrieved 20 January 2020 .   \n\n\u2191 Hagland, M. (26 February 2018). \"A New Era in Network Segmentation?\". Healthcare Innovation. https:\/\/www.hcinnovationgroup.com\/cybersecurity\/article\/13029865\/a-new-era-in-network-segmentation . Retrieved 20 January 2020 .   \n\n\u2191 Hiemstra, H. (2016). \"Influence of Building Structure and Building Content on Residential Fires\". Master's Thesis. Lund University. https:\/\/lup.lub.lu.se\/student-papers\/search\/publication\/8876361 .   \n\n\u2191 Genge, B.; Graur, F.; Haller, P. (2015). \"Experimental assessment of network design approaches for protecting industrial control systems\". International Journal of Critical Infrastructure Protection 11: 24\u201338. doi:10.1016\/j.ijcip.2015.07.005.   \n\n\u2191 Sitting, D.F.; Singh, H. (2016). \"A Socio-Technical Approach to Preventing, Mitigating, and Recovering from Ransomware Attacks\". Applied Clinical Informatics 7 (2): 624\u201332. doi:10.4338\/ACI-2016-04-SOA-0064. PMC PMC4941865. PMID 27437066. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4941865 .   \n\n\u2191 Todinov, M. (2015). \"Reducing Risk through Segmentation, Permutations, Time and Space Exposure, Inverse States, and Separation\". International Journal of Risk and Contingency Management 4 (3): 1\u201321. doi:10.4018\/IJRCM.2015070101.   \n\n\u2191 SL\u00e4nsisalmi, H.; Kivim\u00e4ki, M.; Aalto, P.; Ruoranen, R. (2006). \"Innovation in healthcare: A systematic review of recent research\". Nursing Science Quarterly 19 (1): 66\u201372. doi:10.1177\/0894318405284129. PMID 16407603.   \n\n\u2191 Anell, A.; Glenng\u00e5rd, A.H.; Merkur, S. (2012). \"Sweden health system review\". Health Systems in Transition 14 (5): 1\u2013159. PMID 22894859.   \n\n\u2191 Fridlund, B. (1998). \"Qualitative methods in healthcare research: Some issues related to utilisation and scrutiny\". Care of the Critically Ill 14 (6): 212\u201314. http:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:hj:diva-12022 .   \n\n\u2191 Bowling, A. (2014). Research Methods in Health: Investigating Health and Health Services (4th ed.). Open University Press. ISBN 9780335262748.   \n\n\u2191 Graneheim, U.H.; Lundman, B. (2004). \"Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness\". Nursing Education Today 24 (2): 105\u201312. doi:10.1016\/j.nedt.2003.10.001. PMID 14769454.   \n\n\u2191 Burnard, P. (1996). \"Teaching the analysis of textual data: An experiential approach\". Nursing Education Today 16 (4): 278\u201381. doi:10.1016\/s0260-6917(96)80115-8. PMID 8936234.   \n\n\u2191 Kruse, C.S.; Frederick, B.; Jacobson, T.; Monticore, D.K. (2017). \"Cybersecurity in healthcare: A systematic review of modern threats and trends\". Technology and Health Care 25 (1): 1\u201310. doi:10.3233\/THC-161263. PMID 27689562.   \n\n\u2191 Webb, T.; Dayal, S. (2017). \"Building the wall: Addressing cybersecurity risks in medical devices in the U.S.A. and Australia\". Computer Law & Security Review 33 (4): 559\u2013563. doi:10.1016\/j.clsr.2017.05.004.   \n\n\u2191 Regan, G.; McCaffery, F.; McDaid, K. et al. (2013). \"Medical device standards' requirements for traceability during the software development lifecycle and implementation of a traceability assessment model\". Computer Standards & Interfaces 36 (1): 3\u20139. doi:10.1016\/j.csi.2013.07.012.   \n\n\u2191 Jalali, M.S.; Razak, S.; Gordon, W. et al. (2019). \"Health Care and Cybersecurity: Bibliometric Analysis of the Literature\". Journal of Medical Internet Research 21 (2): e12644. doi:10.2196\/12644. PMC PMC6396074. PMID 30767908. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6396074 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\">https:\/\/www.limswiki.org\/index.php\/Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on cybersecurityLIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 24 February 2020, at 21:55.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 333 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","6fa400ac6b9e1784eed63829a3731260_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Information_technology_and_medical_technology_personnel\u2019s_perception_regarding_segmentation_of_medical_devices_A_focus_group_study skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Information technology and medical technology personnel\u2019s perception regarding segmentation of medical devices: A focus group study<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Objective<\/b>: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Network_segmentation\" class=\"extiw wiki-link\" title=\"wikipedia:Network segmentation\" data-key=\"1f30faaf0ef2ea4238bea33a4e7b2989\">Segmentation<\/a> is one way of improving data protection. The aim of this study was to investigate information technology (IT) and medical technology (MT) personnel\u2019s perception in relation to ongoing segmentation of <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_device\" title=\"Medical device\" class=\"wiki-link\" data-key=\"8e821122daa731f0fa8782fae57831fa\">medical devices<\/a> and IT infrastructure in the healthcare sector. \n<\/p><p><b>Methods<\/b>: Focus group interviews with nine IT and nine MT personnel in a county council in southern Sweden were conducted. The interviews focused on two areas: positive expectations and misgivings. Digital recordings were transcribed verbatim and analyzed using qualitative content analysis. \n<\/p><p><b>Results<\/b>: Responses related to two main areas: <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_security\" title=\"Information security\" class=\"wiki-link\" data-key=\"9eff362d944224ff1d4ffe3a149d7cff\">information security<\/a> and implementation of segmentation. Informants stated that network segmentation would increase the overall level of <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a> for medical devices, addressing both insider and outsider threats. However, it would also increase the need for administration and the need for knowledge of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_device_connectivity\" title=\"Medical device connectivity\" class=\"wiki-link\" data-key=\"2ceff4d24464989037278cac02a0608c\">communication patterns of medical devices<\/a> from the manufacturer\u2019s perspective. \n<\/p><p><b>Conclusion<\/b>: IT and MT personnel in a county council in southern Sweden believed that segmentation would increase cybersecurity but also increase administration and resource needs, which are important opinions to take into consideration. The present study can be used as a model for others to increase awareness of opinions of healthcare organizations. \n<\/p><p><b>Keywords<\/b>: cybersecurity, healthcare technology, patient safety, staff attitudes\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Medical technology (MT) devices\u2014as defined by the European Union's directive on medical devices<sup id=\"rdp-ebb-cite_ref-EUCouncil93_1-0\" class=\"reference\"><a href=\"#cite_note-EUCouncil93-1\">[1]<\/a><\/sup> and information technology (IT) and device developers\u2014serve an increasingly central role in clinical practice, improving patient health, safety, and quality of life. And the number of such <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_device\" title=\"Medical device\" class=\"wiki-link\" data-key=\"8e821122daa731f0fa8782fae57831fa\">medical devices<\/a> that are connected to a network continues to grow.<sup id=\"rdp-ebb-cite_ref-AbernathyDevelop11_2-0\" class=\"reference\"><a href=\"#cite_note-AbernathyDevelop11-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McCafferyRisk10_3-0\" class=\"reference\"><a href=\"#cite_note-McCafferyRisk10-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TulasidasProcess13_4-0\" class=\"reference\"><a href=\"#cite_note-TulasidasProcess13-4\">[4]<\/a><\/sup>\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_device_connectivity\" title=\"Medical device connectivity\" class=\"wiki-link\" data-key=\"2ceff4d24464989037278cac02a0608c\">Networked medical devices<\/a> can also cause substantial harm since they have not historically been designed with a focus on <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a>.<sup id=\"rdp-ebb-cite_ref-ArneyBiomedical11_5-0\" class=\"reference\"><a href=\"#cite_note-ArneyBiomedical11-5\">[5]<\/a><\/sup> Errors that underlie device-related injuries are often categorized into three types: manufacturer-related errors, use or design errors, and user errors.<sup id=\"rdp-ebb-cite_ref-JhaPatient10_6-0\" class=\"reference\"><a href=\"#cite_note-JhaPatient10-6\">[6]<\/a><\/sup> It has been said that the weakest link in the process is the user, who must understand how to configure and use medical technology products correctly to achieve a high level of security in computing infrastructure.<sup id=\"rdp-ebb-cite_ref-IvarssonInfo16_7-0\" class=\"reference\"><a href=\"#cite_note-IvarssonInfo16-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NurseUnder14_8-0\" class=\"reference\"><a href=\"#cite_note-NurseUnder14-8\">[8]<\/a><\/sup>\n<\/p><p>Cybersecurity is today one of the most important security-related challenges for all countries, but its visibility and awareness are still limited to the public, although almost everyone has heard something about it.<sup id=\"rdp-ebb-cite_ref-deBrujinBuild17_9-0\" class=\"reference\"><a href=\"#cite_note-deBrujinBuild17-9\">[9]<\/a><\/sup> There is a substantial security risk posed by outsiders identifying potential interactions between the interconnected elements in <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital_information_system\" title=\"Hospital information system\" class=\"wiki-link\" data-key=\"d8385de7b1f39a39d793f8ce349b448d\">hospital systems<\/a> and computing infrastructures, and then taking advantage of poor cybersecurity to steal <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">medical health records<\/a>, deny access to health services, or cause intentional harm with these.<sup id=\"rdp-ebb-cite_ref-Fischbacher-SmithTheVuln13_10-0\" class=\"reference\"><a href=\"#cite_note-Fischbacher-SmithTheVuln13-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CoventryCyber18_11-0\" class=\"reference\"><a href=\"#cite_note-CoventryCyber18-11\">[11]<\/a><\/sup> For medical devices specifically, it may be possible for hackers to use them in order to gain access to confidential patient data and to reprogram them to send harmful commands.<sup id=\"rdp-ebb-cite_ref-LeavittResearch10_12-0\" class=\"reference\"><a href=\"#cite_note-LeavittResearch10-12\">[12]<\/a><\/sup> This could mean, for example, altering blood groups or test results, or taking control of pumps that regulate the administration of potent drugs.<sup id=\"rdp-ebb-cite_ref-MartinCyber17_13-0\" class=\"reference\"><a href=\"#cite_note-MartinCyber17-13\">[13]<\/a><\/sup> Other harmful activities that could be enacted include disrupting the parameter values used in the scanning protocols, tampering with the radiation exposure levels, causing mechanical disruption, and creating denial-of-service attacks.<sup id=\"rdp-ebb-cite_ref-MahlerKnow18_14-0\" class=\"reference\"><a href=\"#cite_note-MahlerKnow18-14\">[14]<\/a><\/sup>\n<\/p><p>Other researchers have also investigated the subject of cybersecurity in medical devices and have concluded that there is a relationship between the increase in network connectable medical devices and increased cybersecurity risks.<sup id=\"rdp-ebb-cite_ref-SchwartzTheEvolv18_15-0\" class=\"reference\"><a href=\"#cite_note-SchwartzTheEvolv18-15\">[15]<\/a><\/sup> As such, segmenting the network into multiple layers, with security gates such as firewalls in between them, could be an effective way to contain network problems and reduce the impact of a breach in network security.<sup id=\"rdp-ebb-cite_ref-ReichenbergImproving14_16-0\" class=\"reference\"><a href=\"#cite_note-ReichenbergImproving14-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HaglandANew18_17-0\" class=\"reference\"><a href=\"#cite_note-HaglandANew18-17\">[17]<\/a><\/sup> This is comparable to dividing a building into fire zones to delay fire spreading, and enabling firefighting in sections rather than in the building as a whole.<sup id=\"rdp-ebb-cite_ref-HiemstraInfluence16_18-0\" class=\"reference\"><a href=\"#cite_note-HiemstraInfluence16-18\">[18]<\/a><\/sup>\n<\/p><p>Applying the concept of network segmentation to medical devices may reduce the associated risks and prevent intrusion. This entails separating elements such as computers, servers, routers, data, and healthcare personnel into groups, thereby restricting access and better protecting vital services.<sup id=\"rdp-ebb-cite_ref-GengeExperi15_19-0\" class=\"reference\"><a href=\"#cite_note-GengeExperi15-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SittigASocio16_20-0\" class=\"reference\"><a href=\"#cite_note-SittigASocio16-20\">[20]<\/a><\/sup> Segmentation of medical devices is described as a good method for ensuring data security and is recommended to be used to whatever extent is feasible.<sup id=\"rdp-ebb-cite_ref-TodinovReducing15_21-0\" class=\"reference\"><a href=\"#cite_note-TodinovReducing15-21\">[21]<\/a><\/sup> In order to increase understanding of innovations in healthcare organizations, such as segmentation of medical devices, L\u00e4nsisalmi <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-L.C3.A4nsisalmiInnov06_22-0\" class=\"reference\"><a href=\"#cite_note-L.C3.A4nsisalmiInnov06-22\">[22]<\/a><\/sup> argue that health innovations should be investigated from the perspectives of stakeholders, in this case IT and MT personnel. Therefore, the aim of this study was to investigate IT and MT personnel\u2019s perception of segmentation of medical devices and IT infrastructure in the healthcare sector.\n<\/p><p>Highlights of this study:\n<\/p>\n<ul><li> Medical devices can cause substantial harm. <\/li>\n<li> The user of medical devices must understand how to achieve a high level of security. <\/li>\n<li> Hackers might be able to reprogram medical devices to send harmful commands. <\/li>\n<li> One way to reduce the risk of hackers and prevent intrusion is network segmentation. <\/li>\n<li> MT and IT personnel must be positive about and open to the improvement of cybersecurity.<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Design_and_setting\">Design and setting<\/span><\/h3>\n<p>A qualitative methodology design was employed with focus group sessions in order to gain a deeper insight into the views and experiences of IT segmentation. This descriptive study was conducted at Region Sk\u00e5ne, one of the 21 county councils\/regions in Sweden. Region Sk\u00e5ne employs around 34,000 people, mostly in healthcare, and has the main responsibility for public healthcare and medical services in the region.<sup id=\"rdp-ebb-cite_ref-AnellSweden12_23-0\" class=\"reference\"><a href=\"#cite_note-AnellSweden12-23\">[23]<\/a><\/sup> Much of the healthcare is conducted in a high-tech environment, which includes devices such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Monitoring_(medicine)\" title=\"Monitoring (medicine)\" class=\"wiki-link\" data-key=\"53fbc571860de409fc48331d7d9362cb\">patient monitoring<\/a> equipment, <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_image_computing\" title=\"Medical image computing\" class=\"wiki-link\" data-key=\"1a13b99123ccd675ab1ff05108ae0fe5\">medical imaging systems<\/a>, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> instrumentation. A number of cybersecurity-related incidents prompted a decision to implement network segmentation for all devices in Region Sk\u00e5ne. The study was performed during the ongoing implementation of the network segmentation of medical devices.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Informants\">Informants<\/span><\/h3>\n<p>Informants were strategically recruited from IT and MT personnel based on their experience working with medical devices such as patient monitoring equipment, medical imaging systems, and laboratory instrumentation in Region Sk\u00e5ne to ensure maximal variation in socio-demographic data and service positions.<sup id=\"rdp-ebb-cite_ref-FridlundQuali98_24-0\" class=\"reference\"><a href=\"#cite_note-FridlundQuali98-24\">[24]<\/a><\/sup> The following variables were taken into account to ensure a broad selection: age, time in the profession, and competence in (1) segmentation, (2) MT device systems, (3) firewall-based traffic filtering, and (4) IT security.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Ethics\">Ethics<\/span><\/h3>\n<p>The study conformed to the principles outlined in the Declaration of Helsinki, and according to Swedish legislation (SFS 2003:460) no ethical approval was required as the study did not involve patients. The project was approved by the directors of the participating departments. All informants were informed of the aims and procedures of the study through a short written presentation included in an email invitation. Informed consent was considered implied when informants signed up voluntarily for the focus group interviews. The data processing was carried out in accordance to the <a href=\"https:\/\/www.limswiki.org\/index.php\/General_Data_Protection_Regulation\" title=\"General Data Protection Regulation\" class=\"wiki-link\" data-key=\"3f4bdf6f0dcb360b1e79aad8674c2447\">General Data Protection Regulation<\/a> (GDPR).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_collection\">Data collection<\/span><\/h3>\n<p>Three mixed MT and IT focus group interviews<sup id=\"rdp-ebb-cite_ref-BowlingResearch14_25-0\" class=\"reference\"><a href=\"#cite_note-BowlingResearch14-25\">[25]<\/a><\/sup> were held with five to eight informants in each group and included a total of 18 informants (nine MT personnel and nine IT personnel; see Table 1 for demographic and other characteristics). Interviews were conducted in a conference room and took about 90 minutes each. The discussions were digitally recorded (with the informants\u2019 consent) and independently transcribed by a trusted agency.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> Demographic data and other characteristics\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Gender\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\"># of informants\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Women\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Men\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Statistical measure of age\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Result\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mean standard deviation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">52 \u00b1 10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Median (range)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">51 (33\u201367)\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Statistical measure of years<br \/>of professional experience\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Result\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mean standard deviation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 \u00b1 12\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Median (range)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17 (2\u201340)\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Highest level of education\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\"># of informants\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High school (<i>n<\/i>)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">University (<i>n<\/i>)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The interviews were conducted with one researcher who had extensive experience in focus group interviews acting as moderator and a second acting as assistant. A short presentation of network segmentation was held prior to discussions as an introduction to the topic. A rough outline of network segmentation and specifically how it relates to the segmentation of medical devices was also presented (see Figure 1). The interviews focused on positive expectations and misgivings about the solution. Informants were asked two open-ended questions: \u201cWhat are the advantages of segmentation of medical devices?\u201d and \u201cWhat misgivings do you have regarding segmentation of medical devices?\u201d Probing questions were also used.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Johansson_Healthcare2020_8-1.png\" class=\"image wiki-link\" data-key=\"c5ecb60906a59062cf969ae47113ab51\"><img alt=\"Fig1 Johansson Healthcare2020 8-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d9\/Fig1_Johansson_Healthcare2020_8-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Example of rough outline of network segmentation<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Data_analysis\">Data analysis<\/span><\/h3>\n<p>The text from the interviews was analyzed using qualitative content analysis.<sup id=\"rdp-ebb-cite_ref-GraneheimQual04_26-0\" class=\"reference\"><a href=\"#cite_note-GraneheimQual04-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BurnardTeach96_27-0\" class=\"reference\"><a href=\"#cite_note-BurnardTeach96-27\">[27]<\/a><\/sup> Transcribed interviews were first read as a whole by all authors to gain an overall understanding. The text was then divided into meaningful units, which were condensed, coded, and gathered into groups with similar content. These steps were carried out by the authors individually, and then all authors gathered and discussed the findings. These were then abstracted into subcategories that met the study aim. Finally, the subcategories were abstracted into two categories. An example of the analysis process can be found in Table 2. The statements were thoroughly assessed by the authors to ensure that no statements relating to positive expectations or misgivings regarding segmentation were left out due to the condensation of meaning units into codes.\n<\/p><p><br \/>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 2.<\/b> Example of the analysis process, from verbatim meaning unit to category\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Meaning unit\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Condensed\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Code\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Subcategory\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Category\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2026So theoretically, if somebody would have a bad intent, it\u2019s very easy today. Considering that today the risk is greater internally than externally. And it doesn\u2019t have to be malicious, it may be that you make a mistake because you have access to things that you should not have access to. And then that\u2019s really a good thing then, to segment, we clearly get what\u2019s stealing from and we can follow it up in a new way\u2026 (Participant 2)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201c\u2026it doesn\u2019t have to be malicious, of course, you might make a mistake just because you have access to things that you shouldn\u2019t have\u201d (Participant 2)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Improving patient security\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Insider threats\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Information security\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cI think it will require a great deal of resources to get it implemented. I see it as very resource-intensive to implement\u2026 longer lead times and complicated administration. There is, as we stated, not enough resources to handle requests in firewall changes\u2026\u201d (Participant 17)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cI see it [implementation] as very resourceintensive to implement \u2026 longer lead times and complicated administration\u201d (Participant 17)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Resource-intensive project\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Predictions of an increase in costs and administration\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Implementation of segmentation in healthcare\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>Two categories emerged during the analysis of the transcribed interviews: \u201cInformation security\u201d and \u201cImplementation of segmentation in healthcare.\u201d These categories contained six subcategories (Table 3).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 3.<\/b> Distribution of categories and subcategories concerning information technology (IT) and medical technology (MT) personnel views in relation to segmentation of medical devices and IT infrastructure in the healthcare sector (<i>N<\/i> = 18).\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Categories\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Subcategories\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Information security\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Outsider threats\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Implementation of segmentation in healthcare\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>Insider threats<br \/>\nPredictions of an increase in costs and administration<br \/>\nPredictions of required knowledge improvement for the healthcare organization<br \/>\nPossibilities and pitfalls concerning segmentation\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Information_security\">Information security<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Outsider_threats\">Outsider threats<\/span><\/h4>\n<p>Across the three focus groups, attacks originating from outside of the healthcare organization were well known, and the informants had personal experience of such attacks or had heard of them from others. Virus attacks that led to numerous problems and increased work had been experienced. Informants were, on the whole, positive to the possibilities of limiting data virus infections that are offered by segmentation technology, as well as the potential increase in patient safety. They were also positive to the increased protection that segmentation offered to older applications unable to withstand outsider threats due to patching policies or lack of security design.\n<\/p><p>It was stated that segmentation reduces the risk that core systems or patients will be affected during an infection of malicious software or hacker attacks\/intrusions, primarily by restricting access between technical devices.\n<\/p><p>As Informant 3 put it: \u201cThe thing is, that you can divide up and protect \u2026 When you look at virus attacks and other things, that it doesn\u2019t affect different segments but stays within the same segment \u2026 That\u2019s the advantage as I see it.\u201d\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Insider_threats\">Insider threats<\/span><\/h4>\n<p>In relation to insider threats, informants had experienced personnel accidentally forwarding infected emails, which caused great harm. They perceived that segmentation would reduce the risk of users getting hold of restricted information via administrative interfaces that they should not have access to, thereby reducing the probability of them making administrative mistakes. It also would reduce the probability that users, whether by mistake or intention, introduced malicious code into medical devices and\/or core infrastructure. The informants opined that with segmentation, devices would be less likely to be incorrectly configured when connected to the network, thereby reducing the likelihood of failures or connection issues and, in turn, improving patient security. Informant 2 adds: \u201c\u2026 It doesn\u2019t have to be malicious, of course, you might make a mistake just because you have access to things that you shouldn\u2019t have.\u201d\n<\/p><p>However, the informants expressed the benefits of segmentation would be cancelled out, however, due to the risk that so many openings would be needed between segments due to the large number of older devices.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Implementation_of_segmentation_in_healthcare\">Implementation of segmentation in healthcare<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Predictions_of_an_increase_in_costs_and_administration\">Predictions of an increase in costs and administration<\/span><\/h4>\n<p>The informants, in general, predicted an increase in administration and costs for the organization. Because of the higher level of complexity introduced by segmentation, the informants predicted that working with a segmented network would require better planning and documentation. A positive side effect of this would be a better understanding of future costs for the healthcare organization, which would be especially useful for detecting increased costs due to deviating or nonstandard network communication solutions for medical technology. It was also expected that segmentation would lead to better planned maintenance work for medical technology solutions since network communication would be more controlled and structured.\n<\/p><p>The informants assumed that implementing segmentation would increase administrative tasks and the need for resources (i.e., make it time consuming) due to the increased number of firewalls between segments, and openings needed in these. Also, new processes, routines, and administrative tools would be needed, such as automated software. There were, therefore, misgivings that segmentation would lead to longer execution times and that increased planning would be required for network changes.\n<\/p><p>Informant 17 puts it as such: \u201cI see it [implementation] as very resource-intensive to implement \u2026 [with] longer lead times and complicated administration.\u201d\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Predictions_of_required_knowledge_improvement_for_the_healthcare_organization\">Predictions of required knowledge improvement for the healthcare organization<\/span><\/h4>\n<p>The informants perceived that their own knowledge and competence regarding network communication would need to improve, due to a higher level of complexity. Informants had positive expectations of receiving the necessary education and assumed that management felt the same way. However, the informants also stated the importance of informing all relevant parties about the strategy and goal of the network segmentation, including, for example, personnel affiliated with system management and medical technology devices within the healthcare organization.\n<\/p><p>Informant 1 states: \u201cI see an advantage and a disadvantage in the fact that more knowledge is required \u2026 not only from us, but also from our suppliers \u2026 With segmentation, we will need a better understanding of all of our systems.\u201d\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Possibilities_and_pitfalls_concerning_segmentation\">Possibilities and pitfalls concerning segmentation<\/span><\/h4>\n<p>The informants perceived that the segmentation solution provided an opportunity to use the infrastructure to provide restricted communication rights based on actual needs of users and systems, which would provide a high degree of flexibility and thereby better preparedness for new communication requirements. It also presented an easier way to reuse the infrastructure. They also expressed that the design gave an advantage by isolating attacks through the network in an easier way, and that it gave the basis for a more controlled and secure network operations environment by reducing the impact of changes and faults between network segments. Furthermore, they said that dedicated network segments for remote administration would give fewer attack vectors between systems.\n<\/p><p>The informants highlighted the need to communicate overall policies and designs, as well as guidelines for implementation of segmentation, to avoid counterproductive effects. One such effect could be creating too many openings between the segments so that the key benefits of segmentation would be lost.\n<\/p><p>Per Informant 18: \u201cSegmentation is really about trying to minimize consequences \u2026 If we discover that one segment has a virus \u2026 then we shut down the framework, which means that it\u2019s completely isolated.\u201d\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Increased_demands_on_medical_device_manufacturers\">Increased demands on medical device manufacturers<\/span><\/h4>\n<p>The informants had also experienced working with medical device manufacturers who connect to medical devices remotely. It was foreseen that requirements for knowledge, planning ability, and documentation from manufacturers of how their devices work and how they integrate with other devices would increase once segmentation was completed. This would give increased insight into devices, something that informants perceived as currently lacking, both in customers and manufacturers. However, this could also lead to increased cost. \u201c\u2026 And then I think another problem could be that our suppliers, they don\u2019t always know exactly how their applications talk to each other, which ports are used. So that could also be challenging to get right,\u201d adds Informant 15.\n<\/p><p>Segmentation was also expected to give an opportunity for manufacturers to connect to devices for support in more diverse and secure ways. For instance, it would be possible to allow forms of communication other than virtual private network (VPN) without affecting the overall security level. It would also be possible to reduce manufacturer\u2019s access rights to other systems, thus reducing the risk of manufacturer-induced faults (paraphrasing of comments from Informant 5).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>This qualitative study provides insights into the perceptions among IT and MT personnel after a decision to implement network segmentation of medical devices and IT infrastructure in the healthcare sector. Despite the fact that this is a small study, the results corroborated that healthcare systems are complex structures, and medical devices that are integrated into a <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a>\u2019s IT systems are just a minor part of the system as a whole. However, these devices play a critical role in clinical function. If a medical device fails, there are many potential hazards to patient safety. There are many techniques for increasing cybersecurity, including not only network segmentation but also security policies, antivirus software, lifecycle management of operating systems, security patches, drivers, and so on.<sup id=\"rdp-ebb-cite_ref-SchwartzTheEvolv18_15-1\" class=\"reference\"><a href=\"#cite_note-SchwartzTheEvolv18-15\">[15]<\/a><\/sup> Hopefully, the present study may partly provide support and inspire healthcare organizations to develop and communicate a strategy that encompasses all these important parts. Generally, security can be further enhanced if the segmentation is implemented with rigorous and strictly enforced security controls regarding communication between segments. As such, the communication boundaries between segments will not stop cyberattacks entirely unless the network also complies with rules, standards, and requirements that emphasize access control policies using a so called \u201cnext-generation firewall\u201d (NGFW) implementation, which includes additional network device-filtering functionalities such as an intrusion prevention system (IPS), intrusion detection protocol (IDP), activity logging, and web and mail security features. A simple application layer firewall cannot stop camouflaged malware from causing security risks to healthcare facilities.\n<\/p><p>In this study, which focused specifically on expectations and misgivings regarding segmentation, there were two themes that emerged across the categories identified in the analysis, namely information security and implementation of segmentation in healthcare. The informants in the study had experience of insider and outsider threats. In May 2017, a major ransomware attack called WannaCry occurred, which affected 200,000 systems in around 150 countries worldwide.<sup id=\"rdp-ebb-cite_ref-MartinCyber17_13-1\" class=\"reference\"><a href=\"#cite_note-MartinCyber17-13\">[13]<\/a><\/sup> In the U.K., 50 hospitals were affected, patient safety was threatened, and there were calls to highlight cybersecurity in healthcare.<sup id=\"rdp-ebb-cite_ref-MartinCyber17_13-2\" class=\"reference\"><a href=\"#cite_note-MartinCyber17-13\">[13]<\/a><\/sup> In the present study, informants assumed that insider and outsider threats and threats from malicious software, such as ransomware, would decrease with the use of segmentation. Besides the advantages pointed out by the informants, segmentation adds additional security measures to hinder attack vectors. An example is the possibility to deny certain types of network traffic on layer seven of the Open Systems Interconnection (OSI)-stack (i.e., message handling in the application layer), which allows detection of communication patterns. This will decrease the spread of malicious code and denial-of-services threats. Medical devices are regarded to have a low capability of defending themselves against these sorts of threats and will benefit greatly from the added security layers. It could be speculated that if all England's hospitals had implemented segmentation of medical equipment ahead of the WannaCry attack<sup id=\"rdp-ebb-cite_ref-MartinCyber17_13-3\" class=\"reference\"><a href=\"#cite_note-MartinCyber17-13\">[13]<\/a><\/sup>, the impact would probably have been lessened as the dispersion could have been limited. If this were the case, hospitals could have continued to produce care and thus maintained a higher level of patient safety, especially if there had been support for technologies such as an intrusion prevention system (IPS). However, healthcare organizations need to be aware that segmentation, although important, is just one measure to increase cybersecurity; relying on it solely could give a false sense of security.\n<\/p><p>Informants in the study opined that with segmentation, the demand for documentation would increase regarding communication between infrastructure components within and between systems. It was also opined that the number of firewall changes (i.e., allowed network traffic patterns between segments) and the overall workload of the network and system administrators would increase, which, in turn, would lead to technical skills and increased costs. These opinions are important to take into account from a change management perspective as the informants represent a critical success factor in achieving a high cybersecurity level. Under-investing in cybersecurity is not unique for Sweden. It has previously been pointed out that compared to other sectors that spend four to ten percent of their budget, the healthcare sector spends a relatively small amount (one to two percent) of its budget on IT infrastructure. This must be reconsidered so that fundamental patient safety, healthcare technology, and functionality are not compromised.<sup id=\"rdp-ebb-cite_ref-MartinCyber17_13-4\" class=\"reference\"><a href=\"#cite_note-MartinCyber17-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KruseCyber17_28-0\" class=\"reference\"><a href=\"#cite_note-KruseCyber17-28\">[28]<\/a><\/sup> It can be stated that, as healthcare organizations are relying more and more on digital technologies, both regarding medical and other systems, the need for cybersecurity will also become more aligned with other sectors, such as industries and banks.\n<\/p><p>On the basis of the present study, the suggestion of a more structured way of working can be perceived as less flexible with regard to changes, from an administrative point of view, while requiring new processes and new supporting software. On the positive side, segmentation will give better insight into the communication landscape, and a better overview. Historically, cyberattacks have had a significant impact on production, generating high costs for the targeted organization; the healthcare sector must find cost-effective ways to prevent such potentially devastating costs.<sup id=\"rdp-ebb-cite_ref-MartinCyber17_13-5\" class=\"reference\"><a href=\"#cite_note-MartinCyber17-13\">[13]<\/a><\/sup> It must also be assumed that the confidence of stakeholders (patients, manufacturers, and institutions) in the healthcare organization will be reduced after a successful cyberattack.\n<\/p><p>Our results showed a need for improvement in the level of competence, as the informants assumed that a greater knowledge base would be required due to the higher level of complexity associated with segmentation. This corroborates a Swedish study showing that a majority of IT and MT personnel considered recurrent training on IT and MT safety, rules, and regulations, as well as a risk analysis with a focus on patient safety, to be of great importance.<sup id=\"rdp-ebb-cite_ref-IvarssonInfo16_7-1\" class=\"reference\"><a href=\"#cite_note-IvarssonInfo16-7\">[7]<\/a><\/sup> It is also important that the goals and importance of segmentation are communicated throughout the organization, and the processes and guidelines established regarding system placement within the different segments. The healthcare organization must commit to and supply the means for education and training.\n<\/p><p>The informants in this study assumed that segmentation would provide more efficient and flexible ways of working for healthcare personnel since changing network sockets will not result in additional administrative tasks. Also, dedicated segments for remote administration add better control and logging capabilities in relation to external users, such as medical device suppliers providing support. The informants also identified a great need for thorough planning before the implementation of segmentation, to make the administration, configuration, and traffic analysis manageable for the organization. The number of segments increases the amount of hardware required, and it is also important to plan where medical devices are placed to minimize administration between segments.\n<\/p><p>In order to meet the needs of cybersecurity of the future, the present study stated that the knowledge level of medical device manufacturers, in general, needed to be increased in association with segmentation. It has earlier been highlighted that increased collaboration between medical device manufacturers and personnel in the healthcare sector is fundamental to ensure effective protection.<sup id=\"rdp-ebb-cite_ref-WebbBuild17_29-0\" class=\"reference\"><a href=\"#cite_note-WebbBuild17-29\">[29]<\/a><\/sup> One effect of segmentation is that suppliers will gain new possibilities to connect to their medical devices within the healthcare organization, for example, during support cases. Segmentation will not reduce the regulation and security requirements that suppliers need to commit to. This is in agreement with<sup id=\"rdp-ebb-cite_ref-ReganMedical13_30-0\" class=\"reference\"><a href=\"#cite_note-ReganMedical13-30\">[30]<\/a><\/sup> that stated the manufactures had challenges to meet regulation and safety standards including lifecycle management to improve medical device cybersecurity. It is, therefore, important that requirements regarding connection for remote maintenance and support are clearly agreed to during the acquisition process for new medical devices.\n<\/p><p>We believe mixed IT\/MT focus groups increased coverage of the subject area and provided openness in discussions regarding to being better able to extract novel insights and various expert knowledge. However, because of the study\u2019s qualitative nature, the aim was not to generalize the results to the healthcare sector as a whole. However, it is reasonable to assume that the results show a part of the reality of healthcare in one county council as well as give a better understanding of cybersecurity issues. A review on cybersecurity stated that there are very few studies in the area which include human and organizational aspects, as well as strategy and management. This may justify the present study.<sup id=\"rdp-ebb-cite_ref-JalaliHealth19_31-0\" class=\"reference\"><a href=\"#cite_note-JalaliHealth19-31\">[31]<\/a><\/sup> A question we have asked ourselves is whether the outcome would have been different if the interviews had been held in the near future after an attack like the WannaCry ransomware attack.<sup id=\"rdp-ebb-cite_ref-MartinCyber17_13-6\" class=\"reference\"><a href=\"#cite_note-MartinCyber17-13\">[13]<\/a><\/sup> That attack is one more indication that cybersecurity policies must be in place for proactive use in the healthcare section and widely communicated, even if the cost seems high. Further, a large (country-scale) study of the implementation of network segmentation would be of interest for further research, and also a study of the health economic impact of a cyberattack.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>Network segmentation decreases the probability of spreading malicious software and intrusions through a network by introducing barriers between network areas, much in the same way fire zones decreases the probability for fire spreading in a building. Medical devices have historically not had a focus on security features. In this study, it was apparent that MT and IT personnel were positively receptive to the increase in cybersecurity provided by network segmentation but concerned about the increase in the administration that it will entail for medical devices. These opinions are important to take into account to be able to reach the desired increase in cybersecurity. Cybersecurity risks are multifaceted. The present study can be used as a model for other clinical healthcare manufacturers to increase awareness of concerns and opinions in personnel and healthcare organizations in general.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We are grateful to the informants who took part in this research and would also like to thank the Medical Services, Region Sk\u00e5ne, Sweden, for their support.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Conceptualization, B.I. and D.J.; methodology, B.I. and M.C.; formal analysis, D.J., B.I., P.J. and M.C.; investigation, B.I. and M.C.; writing\u2014original draft preparation, B.I., D.J. and M.C.; writing\u2014 review and editing, D.J., B.I., P.J. and M.C. All authors have read and agreed to the published version of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research received no external funding.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-EUCouncil93-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EUCouncil93_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">European Commission (14 June 1993). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eur-lex.europa.eu\/eli\/dir\/1993\/42\/2007-10-11\" target=\"_blank\">\"Council Directive 93\/42\/EEC of 14 June 1993 concerning medical devices\"<\/a>. <i>EUR-Lex<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eur-lex.europa.eu\/eli\/dir\/1993\/42\/2007-10-11\" target=\"_blank\">https:\/\/eur-lex.europa.eu\/eli\/dir\/1993\/42\/2007-10-11<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Council+Directive+93%2F42%2FEEC+of+14+June+1993+concerning+medical+devices&rft.atitle=EUR-Lex&rft.aulast=European+Commission&rft.au=European+Commission&rft.date=14+June+1993&rft_id=https%3A%2F%2Feur-lex.europa.eu%2Feli%2Fdir%2F1993%2F42%2F2007-10-11&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbernathyDevelop11-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbernathyDevelop11_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abernethy, A.P.; Wheeler, J.L.; Bull, J. (2011). \"Development of a health information technology-based data system in community-based hospice and palliative care\". <i>American Journal of Preventative Medicine<\/i> <b>40<\/b> (5 Suppl. 2): S217\u201324. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.amepre.2011.01.012\" target=\"_blank\">10.1016\/j.amepre.2011.01.012<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21521597\" target=\"_blank\">21521597<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+of+a+health+information+technology-based+data+system+in+community-based+hospice+and+palliative+care&rft.jtitle=American+Journal+of+Preventative+Medicine&rft.aulast=Abernethy%2C+A.P.%3B+Wheeler%2C+J.L.%3B+Bull%2C+J.&rft.au=Abernethy%2C+A.P.%3B+Wheeler%2C+J.L.%3B+Bull%2C+J.&rft.date=2011&rft.volume=40&rft.issue=5+Suppl.+2&rft.pages=S217%E2%80%9324&rft_id=info:doi\/10.1016%2Fj.amepre.2011.01.012&rft_id=info:pmid\/21521597&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McCafferyRisk10-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McCafferyRisk10_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McCaffery, F.; Burton, J.; Richardson, I. (2010). \"Risk management capability model for the development of medical device software\". <i>Software Quality Journal<\/i> <b>18<\/b>: 18. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11219-009-9086-7\" target=\"_blank\">10.1007\/s11219-009-9086-7<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Risk+management+capability+model+for+the+development+of+medical+device+software&rft.jtitle=Software+Quality+Journal&rft.aulast=McCaffery%2C+F.%3B+Burton%2C+J.%3B+Richardson%2C+I.&rft.au=McCaffery%2C+F.%3B+Burton%2C+J.%3B+Richardson%2C+I.&rft.date=2010&rft.volume=18&rft.pages=18&rft_id=info:doi\/10.1007%2Fs11219-009-9086-7&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TulasidasProcess13-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TulasidasProcess13_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tulasidas, S.; Mackay, R.; Craw, P. et al. (2013). \"Process of Designing Robust, Dependable, Safe and Secure Software for Medical Devices: Point of Care Testing Device as a Case Study\". <i>Journal of Software Engineering and Applications<\/i> <b>6<\/b> (9A): 1\u201313. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4236%2Fjsea.2013.69A001\" target=\"_blank\">10.4236\/jsea.2013.69A001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Process+of+Designing+Robust%2C+Dependable%2C+Safe+and+Secure+Software+for+Medical+Devices%3A+Point+of+Care+Testing+Device+as+a+Case+Study&rft.jtitle=Journal+of+Software+Engineering+and+Applications&rft.aulast=Tulasidas%2C+S.%3B+Mackay%2C+R.%3B+Craw%2C+P.+et+al.&rft.au=Tulasidas%2C+S.%3B+Mackay%2C+R.%3B+Craw%2C+P.+et+al.&rft.date=2013&rft.volume=6&rft.issue=9A&rft.pages=1%E2%80%9313&rft_id=info:doi\/10.4236%2Fjsea.2013.69A001&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArneyBiomedical11-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArneyBiomedical11_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Arney, D.; Venkatasubramanian, K.K.; Sokolsky, O.; Lee, I. (2011). \"Biomedical devices and systems security\". <i>Conference Proceedings of the IEEE Engineering in Medicine and Biology Society<\/i> <b>2011<\/b>: 2376-9. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FIEMBS.2011.6090663\" target=\"_blank\">10.1109\/IEMBS.2011.6090663<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22254819\" target=\"_blank\">22254819<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Biomedical+devices+and+systems+security&rft.jtitle=Conference+Proceedings+of+the+IEEE+Engineering+in+Medicine+and+Biology+Society&rft.aulast=Arney%2C+D.%3B+Venkatasubramanian%2C+K.K.%3B+Sokolsky%2C+O.%3B+Lee%2C+I.&rft.au=Arney%2C+D.%3B+Venkatasubramanian%2C+K.K.%3B+Sokolsky%2C+O.%3B+Lee%2C+I.&rft.date=2011&rft.volume=2011&rft.pages=2376-9&rft_id=info:doi\/10.1109%2FIEMBS.2011.6090663&rft_id=info:pmid\/22254819&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JhaPatient10-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JhaPatient10_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jha, A.K.; Prasopa-Plaizier, N.; Larizgoitia, I. et al. (2010). \"Patient safety research: an overview of the global evidence\". <i>Quality and Safety in Health Care<\/i> <b>19<\/b> (1): 42\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fqshc.2008.029165\" target=\"_blank\">10.1136\/qshc.2008.029165<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20172882\" target=\"_blank\">20172882<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Patient+safety+research%3A+an+overview+of+the+global+evidence&rft.jtitle=Quality+and+Safety+in+Health+Care&rft.aulast=Jha%2C+A.K.%3B+Prasopa-Plaizier%2C+N.%3B+Larizgoitia%2C+I.+et+al.&rft.au=Jha%2C+A.K.%3B+Prasopa-Plaizier%2C+N.%3B+Larizgoitia%2C+I.+et+al.&rft.date=2010&rft.volume=19&rft.issue=1&rft.pages=42%E2%80%937&rft_id=info:doi\/10.1136%2Fqshc.2008.029165&rft_id=info:pmid\/20172882&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IvarssonInfo16-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-IvarssonInfo16_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-IvarssonInfo16_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ivarsson, B.; Wiinberg, S.; Svensson, M. et al. (2016). \"Information and\/or medical technology staff experience with regulations for medical information systems and medical devices\". <i>Health Policy and Technology<\/i> <b>5<\/b> (4): 383\u201388. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.hlpt.2016.07.008\" target=\"_blank\">10.1016\/j.hlpt.2016.07.008<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+and%2For+medical+technology+staff+experience+with+regulations+for+medical+information+systems+and+medical+devices&rft.jtitle=Health+Policy+and+Technology&rft.aulast=Ivarsson%2C+B.%3B+Wiinberg%2C+S.%3B+Svensson%2C+M.+et+al.&rft.au=Ivarsson%2C+B.%3B+Wiinberg%2C+S.%3B+Svensson%2C+M.+et+al.&rft.date=2016&rft.volume=5&rft.issue=4&rft.pages=383%E2%80%9388&rft_id=info:doi\/10.1016%2Fj.hlpt.2016.07.008&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NurseUnder14-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NurseUnder14_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nurse, J.R.C.; Buckley, O.; Legg, P.A. et al. (2014). \"Understanding Insider Threat: A Framework for Characterising Attacks\". <i>2014 IEEE Security and Privacy Workshops<\/i>: 214-228. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FSPW.2014.38\" target=\"_blank\">10.1109\/SPW.2014.38<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+Insider+Threat%3A+A+Framework+for+Characterising+Attacks&rft.jtitle=2014+IEEE+Security+and+Privacy+Workshops&rft.aulast=Nurse%2C+J.R.C.%3B+Buckley%2C+O.%3B+Legg%2C+P.A.+et+al.&rft.au=Nurse%2C+J.R.C.%3B+Buckley%2C+O.%3B+Legg%2C+P.A.+et+al.&rft.date=2014&rft.pages=214-228&rft_id=info:doi\/10.1109%2FSPW.2014.38&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-deBrujinBuild17-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-deBrujinBuild17_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">de Brujin, H.; Janssen, M. (2017). \"Building Cybersecurity Awareness: The need for evidence-based framing strategies\". <i>Government Information Quarterly<\/i> <b>34<\/b> (1): 1\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.giq.2017.02.007\" target=\"_blank\">10.1016\/j.giq.2017.02.007<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+Cybersecurity+Awareness%3A+The+need+for+evidence-based+framing+strategies&rft.jtitle=Government+Information+Quarterly&rft.aulast=de+Brujin%2C+H.%3B+Janssen%2C+M.&rft.au=de+Brujin%2C+H.%3B+Janssen%2C+M.&rft.date=2017&rft.volume=34&rft.issue=1&rft.pages=1%E2%80%937&rft_id=info:doi\/10.1016%2Fj.giq.2017.02.007&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Fischbacher-SmithTheVuln13-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Fischbacher-SmithTheVuln13_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fischbacher-Smith, D.; Fischbacher-Smith, M. (2013). \"The Vulnerability of Public Spaces: Challenges for UK hospitals under the \u2018new\u2019 terrorist threat\". <i>Public Management Review<\/i> <b>15<\/b> (3): 330\u201343. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F14719037.2013.769851\" target=\"_blank\">10.1080\/14719037.2013.769851<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Vulnerability+of+Public+Spaces%3A+Challenges+for+UK+hospitals+under+the+%E2%80%98new%E2%80%99+terrorist+threat&rft.jtitle=Public+Management+Review&rft.aulast=Fischbacher-Smith%2C+D.%3B+Fischbacher-Smith%2C+M.&rft.au=Fischbacher-Smith%2C+D.%3B+Fischbacher-Smith%2C+M.&rft.date=2013&rft.volume=15&rft.issue=3&rft.pages=330%E2%80%9343&rft_id=info:doi\/10.1080%2F14719037.2013.769851&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoventryCyber18-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CoventryCyber18_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coventry, L.; Branley, D. et al. (2018). \"Cybersecurity in healthcare: A narrative review of trends, threats and ways forward\". <i>Maturitas<\/i> <b>113<\/b>: 48\u201352. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.maturitas.2018.04.008\" target=\"_blank\">10.1016\/j.maturitas.2018.04.008<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29903648\" target=\"_blank\">29903648<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cybersecurity+in+healthcare%3A+A+narrative+review+of+trends%2C+threats+and+ways+forward&rft.jtitle=Maturitas&rft.aulast=Coventry%2C+L.%3B+Branley%2C+D.+et+al.&rft.au=Coventry%2C+L.%3B+Branley%2C+D.+et+al.&rft.date=2018&rft.volume=113&rft.pages=48%E2%80%9352&rft_id=info:doi\/10.1016%2Fj.maturitas.2018.04.008&rft_id=info:pmid\/29903648&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeavittResearch10-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeavittResearch10_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leavitt, N. (2010). \"Researchers Fight to Keep Implanted Medical Devices Safe from Hackers\". <i>Computer<\/i> <b>43<\/b> (8): 11\u201314. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FMC.2010.237\" target=\"_blank\">10.1109\/MC.2010.237<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Researchers+Fight+to+Keep+Implanted+Medical+Devices+Safe+from+Hackers&rft.jtitle=Computer&rft.aulast=Leavitt%2C+N.&rft.au=Leavitt%2C+N.&rft.date=2010&rft.volume=43&rft.issue=8&rft.pages=11%E2%80%9314&rft_id=info:doi\/10.1109%2FMC.2010.237&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MartinCyber17-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MartinCyber17_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-MartinCyber17_13-1\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-MartinCyber17_13-2\">13.2<\/a><\/sup> <sup><a href=\"#cite_ref-MartinCyber17_13-3\">13.3<\/a><\/sup> <sup><a href=\"#cite_ref-MartinCyber17_13-4\">13.4<\/a><\/sup> <sup><a href=\"#cite_ref-MartinCyber17_13-5\">13.5<\/a><\/sup> <sup><a href=\"#cite_ref-MartinCyber17_13-6\">13.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Martin, G.; Martin, P.; Hankin, C. et al. (2017). \"Cybersecurity and healthcare: How safe are we?\". <i>BMJ<\/i> <b>358<\/b>: j3179. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.j3179\" target=\"_blank\">10.1136\/bmj.j3179<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cybersecurity+and+healthcare%3A+How+safe+are+we%3F&rft.jtitle=BMJ&rft.aulast=Martin%2C+G.%3B+Martin%2C+P.%3B+Hankin%2C+C.+et+al.&rft.au=Martin%2C+G.%3B+Martin%2C+P.%3B+Hankin%2C+C.+et+al.&rft.date=2017&rft.volume=358&rft.pages=j3179&rft_id=info:doi\/10.1136%2Fbmj.j3179&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MahlerKnow18-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MahlerKnow18_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mahler, T.; Nissim, N.; Shalom, E. et al.. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/arxiv.org\/abs\/1801.05583\" target=\"_blank\">\"Know Your Enemy: Characteristics of Cyber-Attacks on Medical Imaging Devices\"<\/a>. <i>arXiv<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/arxiv.org\/abs\/1801.05583\" target=\"_blank\">https:\/\/arxiv.org\/abs\/1801.05583<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Know+Your+Enemy%3A+Characteristics+of+Cyber-Attacks+on+Medical+Imaging+Devices&rft.jtitle=arXiv&rft.aulast=Mahler%2C+T.%3B+Nissim%2C+N.%3B+Shalom%2C+E.+et+al.&rft.au=Mahler%2C+T.%3B+Nissim%2C+N.%3B+Shalom%2C+E.+et+al.&rft_id=https%3A%2F%2Farxiv.org%2Fabs%2F1801.05583&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchwartzTheEvolv18-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SchwartzTheEvolv18_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-SchwartzTheEvolv18_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schwartz, S.; Ross, A.; Carmody, S. et al. (2018). \"The Evolving State of Medical Device Cybersecurity\". <i>Biomedical Instrumentation and Technology<\/i> <b>52<\/b> (2): 103-111. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2345%2F0899-8205-52.2.103\" target=\"_blank\">10.2345\/0899-8205-52.2.103<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29558182\" target=\"_blank\">29558182<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Evolving+State+of+Medical+Device+Cybersecurity&rft.jtitle=Biomedical+Instrumentation+and+Technology&rft.aulast=Schwartz%2C+S.%3B+Ross%2C+A.%3B+Carmody%2C+S.+et+al.&rft.au=Schwartz%2C+S.%3B+Ross%2C+A.%3B+Carmody%2C+S.+et+al.&rft.date=2018&rft.volume=52&rft.issue=2&rft.pages=103-111&rft_id=info:doi\/10.2345%2F0899-8205-52.2.103&rft_id=info:pmid\/29558182&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ReichenbergImproving14-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ReichenbergImproving14_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Reichenberg, N. (20 March 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.securityweek.com\/improving-security-proper-network-segmentation\" target=\"_blank\">\"Improving Security via Proper Network Segmentation\"<\/a>. <i>Security Week<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.securityweek.com\/improving-security-proper-network-segmentation\" target=\"_blank\">https:\/\/www.securityweek.com\/improving-security-proper-network-segmentation<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Improving+Security+via+Proper+Network+Segmentation&rft.atitle=Security+Week&rft.aulast=Reichenberg%2C+N.&rft.au=Reichenberg%2C+N.&rft.date=20+March+2014&rft_id=https%3A%2F%2Fwww.securityweek.com%2Fimproving-security-proper-network-segmentation&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HaglandANew18-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HaglandANew18_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Hagland, M. (26 February 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.hcinnovationgroup.com\/cybersecurity\/article\/13029865\/a-new-era-in-network-segmentation\" target=\"_blank\">\"A New Era in Network Segmentation?\"<\/a>. <i>Healthcare Innovation<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.hcinnovationgroup.com\/cybersecurity\/article\/13029865\/a-new-era-in-network-segmentation\" target=\"_blank\">https:\/\/www.hcinnovationgroup.com\/cybersecurity\/article\/13029865\/a-new-era-in-network-segmentation<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+New+Era+in+Network+Segmentation%3F&rft.atitle=Healthcare+Innovation&rft.aulast=Hagland%2C+M.&rft.au=Hagland%2C+M.&rft.date=26+February+2018&rft_id=https%3A%2F%2Fwww.hcinnovationgroup.com%2Fcybersecurity%2Farticle%2F13029865%2Fa-new-era-in-network-segmentation&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HiemstraInfluence16-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HiemstraInfluence16_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Hiemstra, H. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/lup.lub.lu.se\/student-papers\/search\/publication\/8876361\" target=\"_blank\">\"Influence of Building Structure and Building Content on Residential Fires\"<\/a>. <i>Master's Thesis<\/i>. Lund University<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/lup.lub.lu.se\/student-papers\/search\/publication\/8876361\" target=\"_blank\">https:\/\/lup.lub.lu.se\/student-papers\/search\/publication\/8876361<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Influence+of+Building+Structure+and+Building+Content+on+Residential+Fires&rft.atitle=Master%27s+Thesis&rft.aulast=Hiemstra%2C+H.&rft.au=Hiemstra%2C+H.&rft.date=2016&rft.pub=Lund+University&rft_id=https%3A%2F%2Flup.lub.lu.se%2Fstudent-papers%2Fsearch%2Fpublication%2F8876361&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GengeExperi15-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GengeExperi15_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Genge, B.; Graur, F.; Haller, P. (2015). \"Experimental assessment of network design approaches for protecting industrial control systems\". <i>International Journal of Critical Infrastructure Protection<\/i> <b>11<\/b>: 24\u201338. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijcip.2015.07.005\" target=\"_blank\">10.1016\/j.ijcip.2015.07.005<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Experimental+assessment+of+network+design+approaches+for+protecting+industrial+control+systems&rft.jtitle=International+Journal+of+Critical+Infrastructure+Protection&rft.aulast=Genge%2C+B.%3B+Graur%2C+F.%3B+Haller%2C+P.&rft.au=Genge%2C+B.%3B+Graur%2C+F.%3B+Haller%2C+P.&rft.date=2015&rft.volume=11&rft.pages=24%E2%80%9338&rft_id=info:doi\/10.1016%2Fj.ijcip.2015.07.005&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SittigASocio16-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SittigASocio16_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sitting, D.F.; Singh, H. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4941865\" target=\"_blank\">\"A Socio-Technical Approach to Preventing, Mitigating, and Recovering from Ransomware Attacks\"<\/a>. <i>Applied Clinical Informatics<\/i> <b>7<\/b> (2): 624\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4338%2FACI-2016-04-SOA-0064\" target=\"_blank\">10.4338\/ACI-2016-04-SOA-0064<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4941865\/\" target=\"_blank\">PMC4941865<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27437066\" target=\"_blank\">27437066<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4941865\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4941865<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Socio-Technical+Approach+to+Preventing%2C+Mitigating%2C+and+Recovering+from+Ransomware+Attacks&rft.jtitle=Applied+Clinical+Informatics&rft.aulast=Sitting%2C+D.F.%3B+Singh%2C+H.&rft.au=Sitting%2C+D.F.%3B+Singh%2C+H.&rft.date=2016&rft.volume=7&rft.issue=2&rft.pages=624%E2%80%9332&rft_id=info:doi\/10.4338%2FACI-2016-04-SOA-0064&rft_id=info:pmc\/PMC4941865&rft_id=info:pmid\/27437066&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4941865&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TodinovReducing15-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TodinovReducing15_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Todinov, M. (2015). \"Reducing Risk through Segmentation, Permutations, Time and Space Exposure, Inverse States, and Separation\". <i>International Journal of Risk and Contingency Management<\/i> <b>4<\/b> (3): 1\u201321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4018%2FIJRCM.2015070101\" target=\"_blank\">10.4018\/IJRCM.2015070101<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reducing+Risk+through+Segmentation%2C+Permutations%2C+Time+and+Space+Exposure%2C+Inverse+States%2C+and+Separation&rft.jtitle=International+Journal+of+Risk+and+Contingency+Management&rft.aulast=Todinov%2C+M.&rft.au=Todinov%2C+M.&rft.date=2015&rft.volume=4&rft.issue=3&rft.pages=1%E2%80%9321&rft_id=info:doi\/10.4018%2FIJRCM.2015070101&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-L.C3.A4nsisalmiInnov06-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-L.C3.A4nsisalmiInnov06_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">SL\u00e4nsisalmi, H.; Kivim\u00e4ki, M.; Aalto, P.; Ruoranen, R. (2006). \"Innovation in healthcare: A systematic review of recent research\". <i>Nursing Science Quarterly<\/i> <b>19<\/b> (1): 66\u201372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F0894318405284129\" target=\"_blank\">10.1177\/0894318405284129<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16407603\" target=\"_blank\">16407603<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Innovation+in+healthcare%3A+A+systematic+review+of+recent+research&rft.jtitle=Nursing+Science+Quarterly&rft.aulast=SL%C3%A4nsisalmi%2C+H.%3B+Kivim%C3%A4ki%2C+M.%3B+Aalto%2C+P.%3B+Ruoranen%2C+R.&rft.au=SL%C3%A4nsisalmi%2C+H.%3B+Kivim%C3%A4ki%2C+M.%3B+Aalto%2C+P.%3B+Ruoranen%2C+R.&rft.date=2006&rft.volume=19&rft.issue=1&rft.pages=66%E2%80%9372&rft_id=info:doi\/10.1177%2F0894318405284129&rft_id=info:pmid\/16407603&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AnellSweden12-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AnellSweden12_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Anell, A.; Glenng\u00e5rd, A.H.; Merkur, S. (2012). \"Sweden health system review\". <i>Health Systems in Transition<\/i> <b>14<\/b> (5): 1\u2013159. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22894859\" target=\"_blank\">22894859<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sweden+health+system+review&rft.jtitle=Health+Systems+in+Transition&rft.aulast=Anell%2C+A.%3B+Glenng%C3%A5rd%2C+A.H.%3B+Merkur%2C+S.&rft.au=Anell%2C+A.%3B+Glenng%C3%A5rd%2C+A.H.%3B+Merkur%2C+S.&rft.date=2012&rft.volume=14&rft.issue=5&rft.pages=1%E2%80%93159&rft_id=info:pmid\/22894859&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FridlundQuali98-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FridlundQuali98_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fridlund, B. (1998). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:hj:diva-12022\" target=\"_blank\">\"Qualitative methods in healthcare research: Some issues related to utilisation and scrutiny\"<\/a>. <i>Care of the Critically Ill<\/i> <b>14<\/b> (6): 212\u201314<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:hj:diva-12022\" target=\"_blank\">http:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:hj:diva-12022<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Qualitative+methods+in+healthcare+research%3A+Some+issues+related+to+utilisation+and+scrutiny&rft.jtitle=Care+of+the+Critically+Ill&rft.aulast=Fridlund%2C+B.&rft.au=Fridlund%2C+B.&rft.date=1998&rft.volume=14&rft.issue=6&rft.pages=212%E2%80%9314&rft_id=http%3A%2F%2Furn.kb.se%2Fresolve%3Furn%3Durn%3Anbn%3Ase%3Ahj%3Adiva-12022&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BowlingResearch14-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BowlingResearch14_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Bowling, A. (2014). <i>Research Methods in Health: Investigating Health and Health Services<\/i> (4th ed.). Open University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780335262748.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Research+Methods+in+Health%3A+Investigating+Health+and+Health+Services&rft.aulast=Bowling%2C+A.&rft.au=Bowling%2C+A.&rft.date=2014&rft.edition=4th&rft.pub=Open+University+Press&rft.isbn=9780335262748&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GraneheimQual04-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GraneheimQual04_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Graneheim, U.H.; Lundman, B. (2004). \"Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness\". <i>Nursing Education Today<\/i> <b>24<\/b> (2): 105\u201312. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.nedt.2003.10.001\" target=\"_blank\">10.1016\/j.nedt.2003.10.001<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/14769454\" target=\"_blank\">14769454<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Qualitative+content+analysis+in+nursing+research%3A+concepts%2C+procedures+and+measures+to+achieve+trustworthiness&rft.jtitle=Nursing+Education+Today&rft.aulast=Graneheim%2C+U.H.%3B+Lundman%2C+B.&rft.au=Graneheim%2C+U.H.%3B+Lundman%2C+B.&rft.date=2004&rft.volume=24&rft.issue=2&rft.pages=105%E2%80%9312&rft_id=info:doi\/10.1016%2Fj.nedt.2003.10.001&rft_id=info:pmid\/14769454&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BurnardTeach96-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BurnardTeach96_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Burnard, P. (1996). \"Teaching the analysis of textual data: An experiential approach\". <i>Nursing Education Today<\/i> <b>16<\/b> (4): 278\u201381. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fs0260-6917%2896%2980115-8\" target=\"_blank\">10.1016\/s0260-6917(96)80115-8<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/8936234\" target=\"_blank\">8936234<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Teaching+the+analysis+of+textual+data%3A+An+experiential+approach&rft.jtitle=Nursing+Education+Today&rft.aulast=Burnard%2C+P.&rft.au=Burnard%2C+P.&rft.date=1996&rft.volume=16&rft.issue=4&rft.pages=278%E2%80%9381&rft_id=info:doi\/10.1016%2Fs0260-6917%2896%2980115-8&rft_id=info:pmid\/8936234&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KruseCyber17-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KruseCyber17_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kruse, C.S.; Frederick, B.; Jacobson, T.; Monticore, D.K. (2017). \"Cybersecurity in healthcare: A systematic review of modern threats and trends\". <i>Technology and Health Care<\/i> <b>25<\/b> (1): 1\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2FTHC-161263\" target=\"_blank\">10.3233\/THC-161263<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27689562\" target=\"_blank\">27689562<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cybersecurity+in+healthcare%3A+A+systematic+review+of+modern+threats+and+trends&rft.jtitle=Technology+and+Health+Care&rft.aulast=Kruse%2C+C.S.%3B+Frederick%2C+B.%3B+Jacobson%2C+T.%3B+Monticore%2C+D.K.&rft.au=Kruse%2C+C.S.%3B+Frederick%2C+B.%3B+Jacobson%2C+T.%3B+Monticore%2C+D.K.&rft.date=2017&rft.volume=25&rft.issue=1&rft.pages=1%E2%80%9310&rft_id=info:doi\/10.3233%2FTHC-161263&rft_id=info:pmid\/27689562&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WebbBuild17-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WebbBuild17_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Webb, T.; Dayal, S. (2017). \"Building the wall: Addressing cybersecurity risks in medical devices in the U.S.A. and Australia\". <i>Computer Law & Security Review<\/i> <b>33<\/b> (4): 559\u2013563. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.clsr.2017.05.004\" target=\"_blank\">10.1016\/j.clsr.2017.05.004<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+the+wall%3A+Addressing+cybersecurity+risks+in+medical+devices+in+the+U.S.A.+and+Australia&rft.jtitle=Computer+Law+%26+Security+Review&rft.aulast=Webb%2C+T.%3B+Dayal%2C+S.&rft.au=Webb%2C+T.%3B+Dayal%2C+S.&rft.date=2017&rft.volume=33&rft.issue=4&rft.pages=559%E2%80%93563&rft_id=info:doi\/10.1016%2Fj.clsr.2017.05.004&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ReganMedical13-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ReganMedical13_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Regan, G.; McCaffery, F.; McDaid, K. et al. (2013). \"Medical device standards' requirements for traceability during the software development lifecycle and implementation of a traceability assessment model\". <i>Computer Standards & Interfaces<\/i> <b>36<\/b> (1): 3\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.csi.2013.07.012\" target=\"_blank\">10.1016\/j.csi.2013.07.012<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+device+standards%27+requirements+for+traceability+during+the+software+development+lifecycle+and+implementation+of+a+traceability+assessment+model&rft.jtitle=Computer+Standards+%26+Interfaces&rft.aulast=Regan%2C+G.%3B+McCaffery%2C+F.%3B+McDaid%2C+K.+et+al.&rft.au=Regan%2C+G.%3B+McCaffery%2C+F.%3B+McDaid%2C+K.+et+al.&rft.date=2013&rft.volume=36&rft.issue=1&rft.pages=3%E2%80%939&rft_id=info:doi\/10.1016%2Fj.csi.2013.07.012&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JalaliHealth19-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JalaliHealth19_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jalali, M.S.; Razak, S.; Gordon, W. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6396074\" target=\"_blank\">\"Health Care and Cybersecurity: Bibliometric Analysis of the Literature\"<\/a>. <i>Journal of Medical Internet Research<\/i> <b>21<\/b> (2): e12644. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2F12644\" target=\"_blank\">10.2196\/12644<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6396074\/\" target=\"_blank\">PMC6396074<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30767908\" target=\"_blank\">30767908<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6396074\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6396074<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Health+Care+and+Cybersecurity%3A+Bibliometric+Analysis+of+the+Literature&rft.jtitle=Journal+of+Medical+Internet+Research&rft.aulast=Jalali%2C+M.S.%3B+Razak%2C+S.%3B+Gordon%2C+W.+et+al.&rft.au=Jalali%2C+M.S.%3B+Razak%2C+S.%3B+Gordon%2C+W.+et+al.&rft.date=2019&rft.volume=21&rft.issue=2&rft.pages=e12644&rft_id=info:doi\/10.2196%2F12644&rft_id=info:pmc\/PMC6396074&rft_id=info:pmid\/30767908&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6396074&rfr_id=info:sid\/en.wikipedia.org:Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204341\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.905 seconds\nReal time usage: 1.398 seconds\nPreprocessor visited node count: 24143\/1000000\nPreprocessor generated node count: 37191\/1000000\nPost\u2010expand include size: 176749\/2097152 bytes\nTemplate argument size: 56240\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 714.973 1 - -total\n 90.40% 646.343 1 - Template:Reflist\n 75.26% 538.115 31 - Template:Citation\/core\n 66.46% 475.201 26 - Template:Cite_journal\n 9.69% 69.246 4 - Template:Cite_web\n 7.02% 50.177 38 - Template:Citation\/identifier\n 5.34% 38.205 1 - Template:Infobox_journal_article\n 5.09% 36.366 1 - Template:Infobox\n 3.81% 27.227 35 - Template:Citation\/make_link\n 3.37% 24.114 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11420-0!*!0!!en!5!* and timestamp 20200707204340 and revision id 37998\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study\">https:\/\/www.limswiki.org\/index.php\/Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","6fa400ac6b9e1784eed63829a3731260_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d9\/Fig1_Johansson_Healthcare2020_8-1.png"],"6fa400ac6b9e1784eed63829a3731260_timestamp":1594154620,"5ed990f0be9ad0fbc7220eb00577224e_type":"article","5ed990f0be9ad0fbc7220eb00577224e_title":"A security review of local government using NIST CSF: A case study (Ibrahim et al. 2018)","5ed990f0be9ad0fbc7220eb00577224e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study","5ed990f0be9ad0fbc7220eb00577224e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:A security review of local government using NIST CSF: A case study\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nA security review of local government using NIST CSF: A case studyJournal\n \nThe Journal of SupercomputingAuthor(s)\n \nIbrahim, Ahmed; Valli, Craig; McAteer, Ian; Chaudhry, JunaidAuthor affiliation(s)\n \nEdith Cowan University, Embry-Riddle Aeronautical UniversityPrimary contact\n \nEmail: ahmed dot ibrahim at ecu dot edu dot auYear published\n \n2018Volume and issue\n \n74(10)Page(s)\n \n5171\u201386DOI\n \n10.1007\/s11227-019-02972-wISSN\n \n1573-0484Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/link.springer.com\/article\/10.1007\/s11227-018-2479-2Download\n \nhttps:\/\/link.springer.com\/content\/pdf\/10.1007%2Fs11227-018-2479-2.pdf (PDF)\n\n\n\n\n \n This article contains rendered mathematical formulae. You may require the TeX All the Things plugin for Chrome or the Native MathML add-on and fonts for Firefox if they don't render properly for you. \n\n\nContents\n\n1 Abstract \n2 Introduction \n3 The NIST CSF \n4 Methodology \n\n4.1 Determining compliance \n\n\n5 Findings \n6 Recommendations \n\n6.1 Identify \n6.2 Protect \n6.3 Detect \n6.4 Respond \n6.5 Discussing the recommendations \n\n\n7 Related frameworks \n\n7.1 NIST SP 800-53 Rev. 4 \n7.2 Control Objectives for Information and Related Technologies (COBIT5) \n7.3 ISO\/IEC 27001:2013 \n7.4 ISA 62443-2-1:2009 \n7.5 ISA 62443-3-3:2013 \n7.6 Other frameworks \n\n\n8 Conclusion \n9 Acknowledgements \n10 References \n11 Notes \n\n\n\nAbstract \nEvaluating cybersecurity risk is a challenging task regardless of an organization\u2019s nature of business or size, yet it remains an essential activity. This paper uses the National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF) to assess the cybersecurity posture of a local government organization in Western Australia. Our approach enabled the quantification of risks for specific NIST CSF core functions and respective categories and allowed making recommendations to address the gaps discovered to attain the desired level of compliance. This has led the organization to strategically target areas related to their people, processes, and technologies, thus mitigating current and future threats.\nKeywords: NIST Cybersecurity Framework, local government, cybersecurity, risk assessment \n\nIntroduction \nThe National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF)[1] is a risk-based approach to manage risks organizations face from a cybersecurity perspective. Similarly, several frameworks such as NIST SP 800-53[2], COBIT5[3], ISO\/IEC 27001:2013[4], ISA 62443-2-1:2009[5], and ISA 62443-3-3:2013[6] are being used to assess cybersecurity risk from different perspectives, and outcomes are measured using different yardsticks. Often, navigating the various frameworks can be challenging for organizations, especially if such expertise are not present internally. Given the rapidly changing technology and threat landscape, assessing the cybersecurity posture of an organization, regardless of their business or size, is paramount.\nThe main goals of this paper are:\n\n detailing the adoption of the NIST CSF as an assessment tool that targets different levels of the organization, depending on their level of expertise and job function to obtain responses to facilitate assessment;\n quantifying the assessment to reflect severity of actual risk, which in turn enables the organization to effectively address the issues to attain the desired level of compliance; and\n reviewing in detail similar frameworks used in the industry and relevant case studies.\nThe next section provides a background of the NIST CSF and its components. In tandem, we recommend the reader refer to NIST[1] for additional details and strategies for suitable approaches to implement, which would vary from organization to organization. From there, our focus for the paper shifts to demonstrating the application of NIST CSF in a local government organization and providing recommendations based on our findings.\n\nThe NIST CSF \nThe NIST CSF[1] consists of the Framework Core, the Framework Implementation Tiers, and the Framework Profiles. The Framework Core consists of five concurrent and continuous functions: identify, protect, detect, respond, and recover. We designed an assessment tool for our investigation based on these functions, which provided a systematic approach to ascertain the organizations cybersecurity risk management practices and processes.\nThe Framework Implementation Tiers describe the level that an organization's cybersecurity risk management practices comply with the framework. Tiers provide the context and degree to which cybersecurity risks are managed, as well as the extent to which business needs are considered in cybersecurity risk management. The assessment tool enabled the determination of the organization's Current Tier based on various internal and external factors such as their risk management practices, threat environment, legal and regulatory requirements, business\/mission objectives, and organizational constraints. Organizations should also determine the desired tier, provided it is feasible to implement, reduces cybersecurity risks, and meets organizational goals. The following are descriptions of the tier levels[1]:\n\n Tier-1 (Partial): Risk management practices are not formalized and managed in an ad hoc manner, lack awareness of cybersecurity risks organization-wide, and do not have processes in place to collaborate with external entities.\n Tier-2 (Risk Informed): Risk management practices are formalized but not integrated organization-wide. Cybersecurity activities are prioritized based on risks, with adequate means to perform related duties, and with informal means to communicate cybersecurity information internally and externally.\n Tier-3 (Repeatable): Risk management practices are formalized and policies are in-place and adaptable to cyber threats. An organization-wide approach is required to manage cybersecurity, with skilled and knowledgeable personnel required to rapidly respond, understand dependencies, and understand the role of external partners.\n Tier-4 (Adaptive): Cybersecurity practices are based on lessons learned and predictive indicators, with continuous improvement, adaptability, and timely response. An organization-wide approach is required to manage cybersecurity risks. Cybersecurity is part of the organizational culture, and the organization actively shares with external partners.\nThe Framework Profile represents the outcomes based on the business needs the organization characterized from the Framework Core and determined using the assessment tool. Consequently, a \"current profile\" (the \u201cas is\u201d state) and a \"target profile\" (the \u201cto be\u201d state) can be used to identify opportunities for improving the cybersecurity of the organization.[1] Framework profiles can be determined based on particular implementation scenarios, and therefore, the gap between the current profile and the target profile would vary as per scenario. In this paper, a local government-specific approach to CSF was adapted. However, industry-specific tailoring may be performed for the CSF.\n\nMethodology \nThe NIST CSF allowed us to design an assessment tool targeted at three levels of participants within the organization: executive, management, and technical. The rationale was to ascertain organization-wide understanding of cybersecurity risks. Hence, the assessment tool comprised of questions addressing the requirements outlined as per the NIST CSF.\nThe questions were selected based on the nature and relevance to the level of the participant. This is because the NIST CSF comprised of questions that were both technical and non-technical. Therefore, it would have been unrealistic to expect deep knowledge of technical operations or implementation level details from a policy level executive.\nIn order to assist us determine a baseline (i.e., the desired tier), additional questions were included in the assessment tool to determine the nature of the organization and its business. This was then followed by the remaining requirements comprised in the NIST CSF.\n\nDetermining compliance \nThe compliance for each measure was based on the responses provided by the participants. They were graded as complaint, partially compliant, or non-compliant, and each was assigned scores of either 10, 5, or 0, respectively, for each core function\u2019s subcategory. Any subcategory that was not applicable based on the desired tier level was excluded from the compliance score calculation.\nGiven the number of security requirements for each core function\u2019s subcategory is N, then the number of applicable requirements in each subcategory given the desired tier level is N\u2032. Therefore, the total compliance score C for each core function\u2019s category can be defined as:\n\r\n\n\n \n \n \n \n \n \n \n \n C\n =\n \n \n \n Σ\n R\n \n \n Σ\n \n \n N\n \n ′\n \n \n ×\n 10\n \n \n \n \n \n \n \n \n \n \n \n {\\displaystyle {\\begin{array}{l}{C={\\frac {\\Sigma R}{\\Sigma {N^{\\prime }\\times 10}}}}\\\\\\end{array}}}\n \n \nwhere R is the compliance score for each category of the respective core function.\n\r\n\nAdditionally, a detailed document audit was conducted on existing policies and procedures. The information technology (IT) infrastructure (internal, remote locations, and cloud) were reviewed, and a detailed internal vulnerability assessment was also conducted during our investigation.\n\nFindings \nThe responses provided by the executive, management, and technical participants gave insight into the organization\u2019s cybersecurity posture. Table 1 shows the summary of the compliance of NIST CSF assessment. The compliance scores were determined based on the previously presented equation.\n\n\n\n\n\n\n\nTable 1. NIST CSF compliance matrix\n\n\nFunction\n\nCategory\n\nCompliance (%)\n\nTotal (%)\n\n\nIdentify (ID)\n\nAsset management (ID.AM)\n\n33\n\n36\n\n\nBusiness environment (ID.BE)\n\n75\n\n\nGovernance (ID.GV)\n\n25\n\n\nRisk assessment (ID.RA)\n\n25\n\n\nRisk management strategy (ID.RM)\n\n0\n\n\nProtect (PR)\n\nAccess control (PR.AC)\n\n60\n\n45\n\n\nAwareness and training (PR.AT)\n\n70\n\n\nData security (PR.DS)\n\n50\n\n\nInformation protection processes\r\nand procedures (PR.IP)\n\n20\n\n\nMaintenance (PR.MA)\n\n75\n\n\nProtective technology (PR.PT)\n\n38\n\n\nDetect (DE)\n\nAnomalies and events (DE.AE)\n\n0\n\n25\n\n\nSecurity continuous monitoring (DE.CM)\n\n43\n\n\nGovernance (DE.DP)\n\n25\n\n\nRespond (RS)\n\nResponse planning (RS.AM)\n\n0\n\n38\n\n\nCommunications (RS.CO)\n\n88\n\n\nAnalysis (RS.AN)\n\n0\n\n\nMitigation (RS.MI)\n\n0\n\n\nImprovements (RS.IM)\n\n100\n\n\nRecover (RC)\n\nRecovery planning (RC.RP)\n\n100\n\n100\n\n\nImprovements (RC.IM)\n\n100\n\n\nCommunications (RC.CO)\n\n100\n\n\n\nFor the \"identify\" core function, the organization scored 36%. Their ability to track assets centrally, keep management informed, and understand operational risks from a cybersecurity perspective was limited, while a strategy to manage such risks did not exist. However, the organization understood its business well and was able to set priorities to support risk management decisions.\nAccess to physical\/virtual assets were through authorization and well-defined processes. The staff were trained and informed adequately of information security-related duties and responsibilities. Certain aspects of data security related to confidentiality and availability were done reasonably well; however, assuring integrity of data needed improvement. Similarly, local maintenance and remote maintenance of IT infrastructure were carried out in a manner consistent to policies and procedures. However, relevant policies, processes, and procedures, as well as technology to assist the protection of information systems and relevant assets, were lacking. Therefore, in aggregate, the organization scored 45% compliance for the \"protect\" core function.\nThe organization scored weakest in the detection of cybersecurity incidents, with a score of 25%. Although certain monitoring activities were in place to track physical security and malicious code, timely detection of anomalous activities and detection processes were lacking or non-existent.\nDespite the lack of a specific response plan to respond to cybersecurity events, the organization had measures in place to report incidents and coordinate activities to respond adequately, which resulted in a 38% compliance score for the \"respond\" core function. These practices are updated from time to time; however, mechanism to perform post-incident analysis or to mitigate future cybersecurity events has not been implemented presently.\nInterestingly, the organization was well prepared to deal with recovery and resumption of core services after a cybersecurity event. The recovery plans in place are tested, updated, and improved periodically, thus receiving full compliance for the \"recover\" core functionality of the framework.\n\nRecommendations \nBased on the findings, the following recommendations were made with respect to each core function of the NIST CSF.\n\nIdentify \n(a) Establish a central inventory of assets, including physical devices and systems, software, and external systems, with all required information, and prioritize based on classification, criticality, and business value.\n(b) Identify the organizations role in the supply chain (i.e., producer-consumer model) as it captures and retains public data, collects revenue, and provides services to its stakeholders.\n(c) Establish an information security policy and reference relevant federal and state policies regarding cybersecurity to ensure legal and regulatory requirements are understood and managed.\n(d) Identify and prioritize threats and vulnerabilities, both internal and external, to determine cybersecurity risks to the organization's operations, assets, and individuals.\n(e) Establish risk management processes that are managed and agreed to by stakeholders to support operational risk decisions.\nProtect \n(a) Strengthen the access control policy and procedures for organization-wide assets that require both physical and remote access.\n(b) Sensitize and increase awareness about cybersecurity throughout the workforce more comprehensively, and provide adequate cybersecurity training based on roles and responsibilities. In this regard, clearly describe cybersecurity roles and responsibilities for relevant staff and external stakeholders.\n(c) Enforce required provisions for data security in the policy, and implement data-at-rest and data-in-transit security, as well as integrity-checking mechanisms to ensure confidentiality, integrity, and availability of information and data.\n(d) Establish required policies, processes, and procedures to manage protection of information assets. This include the establishment of lacking policies and processes, particularly for configuration management, data destruction, and physical operating environment; identification of security baselines; SDLC for system management; and formulation of vulnerability, response, and recovery plans.\n(e) Strengthen processes that control and log remote access to organizational assets by external maintenance contractors.\n(f) Establish a central log of organization-wide information systems and devices, establish removable media policy, and strengthen network segregation to protect communication and control networks.\nDetect \n(a) Determine baselines for network operations and data flows, and implement appropriate activities to detect and analyze events based on event data aggregated from multiple sources and sensors. Determine incident impact and threshold to prepare and allocate resources appropriately.\n(b) Implement tools to monitor cyber and physical environments to detect unauthorized mobile code, external service provider activities, and unauthorized access. Perform organization-wide vulnerabilities regularly.\n(c) Outline detection requirements in information security policy, and continuously improve these processes to ensure timely and adequate awareness of anomalous events.\nRespond \n(a) Establish processes and procedures to respond to cybersecurity events in a timely manner.\n(b) Define cybersecurity roles and responsibilities in information security policy to ensure activities are coordinated for internal and external stakeholders, including law enforcement, in response to cybersecurity events.\n(c) Implement required cybersecurity event notification and detection systems to ensure adequate information is available to analyze and understand the impact to support recovery activities.\n(d) Implement required cybersecurity controls to detect, report, and contain incidents to prevent escalation of an incident, mitigate its effect, and eradicate the incidents.\nDiscussing the recommendations \nEach of the above recommendations also had specific internal stakeholder(s) identified to indicate ownership and responsibility for addressing the issues associated. Consequently, the organization was then able to develop strategies to address the issues identified and assign specific tasks to individuals. For this purpose, the organization established an internal document using Microsoft Power BI[7] (typically referred to as a Power BI site) to track and visualize the status of the NIST CSF assessment (Fig. 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Microsoft Power BI Internal Site for tracking, visualizing, and reporting NIST CSF assessment findings, courtesy of the participating local government organization\n\n\n\nThe Power BI site facilitated transparency, visibility, and central reporting throughout the organization. Intuitively, this resulted in a rapid and responsive drive for the organization to address and prioritize issues based on severity and cost, with the goal of achieving Tier-2 compliance.\nFurthermore, a desire to achieve a higher compliance level such as Tier-3 was expressed. Such aspiration is encouraged, however, with caution. Even though a higher level of compliance will improve the cybersecurity posture of the organization, it will also affect other aspects such as resources and cost. For example, contrast the risk management process between Tier-2 and Tier-3 as defined in the NIST CSF[1]:\n\n(a) Implementation of risk management practices are not mandatory in Tier-2, whereas these have to be implemented as organization-wide policies in Tier-3. Thus, Tier-3 organizations should have the procedures, processes, technology, and human resources to implement relevant policies.\n(b) The cybersecurity activities\u2019 priorities are updated in a passive nature in Tier-2 as opposed to regular active updates and constant re-evaluation of priorities for Tier-3 compliance. To acquire such capability, an organization requires adequate technology, skilled human resources, and relevant policies that would enable keeping pace with the changes in the technology and threat landscape.\nIn addition to the two points highlighted above, considering both the integrated risk management program and external participation[1], significant investment in resources and human skills development or acquisition is needed to make the transition from Tier-2 to Tier-3. Moreover, this should only be considered carefully based on the organization\u2019s business requirements, strategic objectives, budget, risk appetite, and current and future threats.\n\nRelated frameworks \nThe diversity and complexity of information technology (IT) system components have increased significantly in recent years. Consequently, in order for businesses to adequately secure these systems, several standards and frameworks have been developed.[8] Such frameworks need to be applicable to all manner of business sectors, be they government or private, enterprise or small-business.\nSince NIST CSF can be considered a high-level abstraction of related frameworks, it provides references to other related frameworks for specific implementation guidelines. These referenced frameworks include:\n\n NIST SP 800-53 Rev. 4\n Control Objectives for Information and Related Technologies (COBIT5)\n ISO\/IEC 27001:2013\n ISA 62443-2-1:2009\n ISA 62443-3-3:2013\nThese are further described below.\n\nNIST SP 800-53 Rev. 4 \nNIST SP 800-53[2] revisions are made according to changes in responsibilities under the Federal Information Security Management Act (FISMA), Public Law (P.L.) 107-347. Revision 4 of this framework consists of five functions (identify, protect, detect, respond, and recover), 22 categories, and 98 subcategories. This framework utilizes a four-tier security model (partial, risk informed, repeatable, and adaptive) and a seven-step process (prioritize and scope, orient, create a current profile, conduct a risk assessment, create a target profile, determine, analyze and prioritize gaps, and implement action plan). It focuses on assessing the current situation by determining how to assess security, how to consider risk, and how to resolve the security threats.\nTable 2 provides a summary of useful examples of how the NIST SP 800-53 framework has been applied in practice.\n\n\n\n\n\n\n\nTable 2. Summary of case studies for NIST SP 800-53\n\n\nCase study\n\nDescription\n\n\nMaroochy water services cyber attack against critical infrastructure in 2000[9]\n\nDisgruntled former employee used insider knowledge and stolen configurations and equipment to release more than one million liters of untreated sewerage water resulting in considerable environmental damage and prosecution by the Environmental Protection Agency. The case study revealed that the application of CSF controls would have mitigated the cyber attack.\n\n\nIntel\u2019s high-risk IT business units\u2019 pilot project[10]\n\nIntel IT\u2019s Office and Enterprise business units, considered to be high-level risk environments, were the test bed for a pilot project to test the effectiveness of the NIST SP 800-53. The benefits of using this framework were realized within a short time-frame, with coherent use of risk management technologies across the business model, improved identification of strengths and weaknesses, and more efficient assessments of security priorities. As a result of the pilot project, Intel IT planned to expand the framework\u2019s implementation throughout their business infrastructure.\n\n\nCybersecurity framework implementation at the University of Chicago[11]\n\nThe University of Chicago used the framework to establish cybersecurity protection for its Biological Services Division (BSD). The four-part implementation consisted of identifying the initial state of cybersecurity processes, assessment of the initial threat landscape, determination of the desired target status, and creation of a roadmap to establish and monitor progress. This resulted in improved identification of security requirements and target objectives, better development and maintenance of departmental processes to achieve these objectives, more long-term security solutions in a cost-effective manner, and improved information-sharing and good work practices across departments with different cybersecurity requirements.\n\n\nHow the University of Pittsburgh is using the NIST Cybersecurity Framework[12]\n\nThe University of Pittsburgh used NIST SP 800-53 to implement an IT security package that would cater for diversified needs while enabling collaboration between different departments, accommodate a wide variety of information types and sensitivities, and encompass third-party contractors on an ad hoc basis. NIST SP 800-53 enabled these goals to be met through the streamlining of existing practices and improving documentation. The scalable nature of NIST CSF was applicable to the differing scope and IT requirements of each department within the University.\n\n\nSIEM-based framework for security controls automation[13]\n\nThe potential of using SIEM technology is investigated with the aim of maximizing security-control automation. For the security controls identified in NIST SP 800-53, approximately 30% of these controls were considered as having the capability of automation control. The cost of implementing a SIEM-based framework for security-control automation would be quickly recouped within a short time compared to the reduced employee hours required to monitor an infrastructure the size of a local government organization.\n\n\nRecommendations for information security awareness training for college students[14]\n\nA survey largely based on NIST SP 800-50 was designed to assess information security awareness among students at the business college of a mid-sized university in New England. The survey found that less than one-quarter of the participants had undertaken any form of information security awareness training (ISAT), and only two of the 68 had enrolled in university-provided training. ISAT of employees in local government is an integral part of a well-implemented cybersecurity infrastructure. Any cybersecurity review needs to ascertain current levels of information security awareness to gauge whether existing training is effective or deficient. The training needs to be regularly updated as new vulnerabilities and threats continually develop in this field.\n\n\n\nControl Objectives for Information and Related Technologies (COBIT5) \nCOBIT5[3] is a business CSF designed for the governance and maintenance of enterprise IT systems. It consists of five domains and 37 processes in-line with the responsibility areas of plan, build, run, and monitor. COBIT5 is aligned and coordinated with other recognized IT standards and good practices, such as NIST, ISO 27000, COSO, ITIL, BiSL, CMMI, TOGAF, and PMBOK. It is built around the following considerations:\n\n meeting stakeholder expectations.\n controlling enterprise process end-to-end\n working as a single integrated framework\n recognizing that \u201cmanagement\u201d and \u201cgovernance\u201d are two different things\nISO\/IEC 27001:2013 \nISO\/IEC 27001:2013[4] is an international information security standard published by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), which originated from the British Standard, BS 7799. This framework consists of 114 controls in 14 groups describing the requirements needed to design and implement an information security management system (ISMS). Version 2, released in 2013, replaces the original 2005 version. It is a standard that should be implemented by all businesses where information security is a critical factor, but in particular, applies to software development, managed service providers\/hosting services providers, IT, banking and insurance, information management, government agencies and their service providers, and e-commerce merchants.[4]\nTable 3 provides a summary of useful examples of how the ISO\/IEC 27001:2013 framework has been applied in practice.\n\n\n\n\n\n\n\nTable 3. Summary of case studies for ISO 27001:2013\n\n\nCase study\n\nDescription\n\n\nThames Security Shredding (TSS) Ltd.[15]\n\nTSS specializes in the collection and destruction of confidential documentation on a commercial scale. Maintaining information security is, therefore, a critical process to protect their clients\u2019 identity and to ensure compliance with the UK Data Protection Act 1998. Certification to the ISO\/IEC 27001 standard was seen as an integral part of the implementation and maintenance of world-class customer-centric security controls that would satisfy existing and prospective customers\u2019 needs and allow for rapid growth in the business. ISO\/IEC 27001 certification resulted in an improved attitude and awareness of their staff towards information-security-related issues. A risk-based business continuity plan was used to minimize the impact of any potential security breaches. The certification allowed documentation to be continually updated and improved as corrective actions were taken.\n\n\nFredrickson International[16]\n\nDebt collection is a sector which\u2014like banking, finance, telecommunications, and local government\u2014is coming under increasing scrutiny and regulation. Fredrickson International is a debt collection agency that lists a central government department, as well as several U.K. financial institutions and FTSE 100 companies, among their clients. Since attaining ISO\/IEC 27001 certification, Fredrickson has achieved higher levels of security awareness throughout its departments, staff, and employees. Security audits have become more streamlined, and customers were given the confidence that Fredrickson was conducting international best practice when it came to information security.\n\n\nLegal Ombudsman[17]\n\nThe office of the Legal Ombudsman for England and Wales was established to simplify the process by which members of the public, small businesses, charities, clubs, trusts, etc. could resolve complaints about legal practitioners. To improve its customer service, information security practices conducted by the office needed to show that greater information security awareness had been established, diligence and compliance in handling sensitive information were in place, and that an assurance framework was aligned with global best practice. ISO\/IEC 27001 certification helped the Legal Ombudsman for England and Wales to provide clients with the confidence and reassurance that it was conducting its work by the highest work standards. A better understanding of the information security among its staff led to a reduction in risk and an increase in productivity.\n\n\nSVM Cards Europe[18]\n\nSVM is a leading provider of gift card, voucher, e-code, reward code, and similar promotional and benefit schemes throughout Europe. SVM required secure business processes, improved internal organization, and increased information protection. It also sought greater tender and competitive advantage. With ISO\/IEC 27001 certification, SVM observed that processes became more of a lifestyle than strictly about security only, which resulted in less downtime, instigated a stronger organizational structure, improved on its ability to win new contracts, and have greater confidence that their information security processes were working properly.\n\n\nInfoView Technologies[19]\n\nInfoView Technologies, a Queensland-based data analytics company, required a business model that met state government requirements, improved data security understanding, became more professional, improved its business culture, and was able to sustain and continuously improve its information security management, systems, and policies. These goals were achieved through ISO\/IEC 27001 accreditation, after which InfoView Technologies were able to gain increased market access, meet compliance requirements of the Queensland state government, reduce risk, become more competitive, and streamline its practices and business culture.\n\n\nCapgemini[20]\n\nCapgemini is the largest IT services company in Europe and a global leader in its multiple domains of services. Operating in more than 40 countries, and over 100 languages, Capgemini\u2019s business model needed to transcend national and cultural boundaries. Systems were required to be robust to avoid losing business and maintain competitiveness. Protection of client assets and resources was deemed a priority to assure confidentiality, integrity, and availability. Through ISO\/IEC 27001 certification, Capgemini was able to ensure improved security within its departments and for its clients, enhance security awareness in its staff and employees, and provide more efficient and streamlined documentation and reporting procedures. Standards certification needed to be applicable within the global marketplace and remain pertinent regardless of cultural differences.\n\n\nCostain[21]\n\nCostain, a U.K.-based engineering and construction group, has contributed to the construction of significant projects worldwide. Obtaining standards certifications was seen as the correct path to achieve improvements in several internal processes. Such goals required the implementation of several standards, such as quality management standard (ISO 9001), environmental management (ISO 14001), health and safety (BS OHSAS 18001), collaborative business relationships (BS 11000), information security management (ISO\/IEC 27001), and business continuity management (ISO 22301). Through the enactment of multiple standards, Costain was able to improve several areas of their business to the benefit of their internal and external customers.\n\n\n\nISA 62443-2-1:2009 \nISA 62443-2-1:2009[5] is an International Standards on Auditing (ISA) standard covering the elements required to develop an industrial automation control system - security management system (IACS-SMS). It consists of three categories, three element groups, and 22 elements. The framework is the first of four ISA policy and procedure products that identifies the essentials necessary to establish an effective cybersecurity management system (CSMS). However, the step-by-step approach as to how this is achieved is company-specific and according to their own business culture. These essentials addressed by this standard include:\n\n analyzing risk\n addressing risk with the CSMS\n monitoring and improving the CSMS\nISA 62443-3-3:2013 \nISA 62443-3-3:2013[6] is an International Standards on Auditing (ISA) standard covering the elements required for cybersecurity controls of industrial control systems (ICS). It consists of seven foundation requirements and 51 system requirements.\nISA 62443-3-3:2013 is the third of three ISA systems products that outlines system security requirements and security levels.[6]\n\nOther frameworks \nIn addition to the above, other frameworks used in the industry include the following:\n\n The Committee of Sponsoring Organizations of the Treadway Commission's (COSO) \u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200bEnterprise Risk Management standard is designed jointly by five leading associations, with the aim of integrating strategy and performance.[22]\n The Council on CyberSecurity (now Center for Internet Security) CIS Controls consist of a prioritized set of actions, originally developed by the SANS Institute, to protect assets from cyber attack.[23]\n ISF Standard of Good Practice for Information Security is a standard aimed at providing controls and guidance on all aspects of information security.[24]\n ETSI Cyber Security Technical Committee (TC Cyber) was developed to improve standards within the European telecommunications sector.[25] \n The Sherwood Applied Business Security Architecture (SABSA) Enhanced NIST Cybersecurity (SENC) project enhances the five core levels of the NIST CSF into a SABSA model consisting of a six-level security architecture.[26]\n The IASME Consortium's Cyber Essentials is an information assurance standard based on ISO 27000 but aimed at small businesses.[27]\n The IETF's RFC 2196 Site Security Handbook represents a guide on how to develop computer security policies and procedures.[28]\n Health Information Trust Alliance's HITRUST CSF is the first IT security CSF designed specifically for the healthcare sector. It is based on existing NIST standards and is aimed at healthcare and information security professionals.[29]\n North American Electric Reliability Corporation Critical Infrastructure Protection (NERC-CIP), Version 5 is a set of requirements needed to secure the assets of the North American bulk electric system.[30]\n Open Security Architecture (OSA) is a free community-owned resource of advice on the selection, design, and integration of devices required to provide security and control of an IT network.[31]\n Good Practice Guide 13 (GPG13) is a U.K. government CSF related to Code of Connection (CoCo) compliance for businesses to secure IT systems.[32]\nConclusion \nIn this paper we have used the NIST CSF to evaluate the cybersecurity risks of a local government organization in Western Australia. Our approach can be used to derive measurable metrics for each Framework Core function and respective categories, thus enabling the organization to ascertain the cybersecurity preparedness to actual risk.\nOur findings suggest that evaluating the desired tier compliance to the NIST CSF helps identify the specific people, processes, and technology areas that require improvement (i.e., gaps), which directly influence threat mitigation. The application of CSF helped us understand the current security context of the organization while identifying the risks and future growth areas to improve. While higher tier compliance may be desired, we have also recommended that the organization\u2019s business requirements, strategic goals, budget, risk appetite, and current and future threats be considered carefully.\nFurthermore, as we have presented several related frameworks, navigating such frameworks for self-assessment can be challenging\u2014often not intended by design\u2014but not impossible. We have observed that the NIST CSF offers an advantage over other frameworks in this regard. However, there is still room for developing additional tools that would simplify the implementation process and speed up adoption.\nTherefore, our future work will aim to improve the current assessment tool we have used, with a focus of making it adaptable and accessible to a wider audience and measurable for accurate quantification of cyber preparedness.\n\nAcknowledgements \nWe would like to thank the Western Australia local government organization for sharing their case study for this research. We would also like to thank their staff for their support and cooperation during the assessment.\n\nReferences \n\n\n\u2191 1.0 1.1 1.2 1.3 1.4 1.5 1.6 National Institute of Standards and Technology. \"Cybersecurity Framework\". https:\/\/www.nist.gov\/cyberframework .   \n\n\u2191 2.0 2.1 National Institute of Standards and Technology (December 2014). \"Assessing Security and Privacy Controls in Federal Information Systems and Organizations\" (PDF). NIST Special Publication 800-53A, Revision 4. https:\/\/nvlpubs.nist.gov\/nistpubs\/SpecialPublications\/NIST.SP.800-53Ar4.pdf . Retrieved 01 February 2018 .   \n\n\u2191 3.0 3.1 ISACA. \"COBIT 5\". https:\/\/cobitonline.isaca.org\/ . Retrieved 01 February 2018 .   \n\n\u2191 4.0 4.1 4.2 International Standards Organization. \"ISO\/IEC 27001:2013 - Information technology \u2014 Security techniques \u2014 Information security management systems \u2014 Requirements\". https:\/\/www.iso.org\/standard\/54534.html . Retrieved 01 February 2018 .   \n\n\u2191 5.0 5.1 ISA (13 January 2009). \"Security for industrial automation and control systems, Part 2-1: Establishing an industrial automation and control systems security program\" (PDF). ANSI\/ISA-62443-2-1 (99.02.01)-2009. http:\/\/www.icsdefender.ir\/files\/scadadefender-ir\/paygahdanesh\/standards\/ISA-62443-2-1-Public.pdf . Retrieved 13 March 2018 .   \n\n\u2191 6.0 6.1 6.2 ISA (12 August 2013). \"Security for industrial automation and control systems, Part 3-3: System security requirements and security levels\" (PDF). ANSI\/ISA-62443-3-3 (99.03.03)-2013. http:\/\/www.icsdefender.ir\/files\/scadadefender-ir\/paygahdanesh\/standards\/ISA-62443-3-3-Public.pdf . Retrieved 13 March 2018 .   \n\n\u2191 Microsoft. \"Microsoft Power BI\". https:\/\/powerbi.microsoft.com\/en-us\/ . Retrieved 12 April 2018 .   \n\n\u2191 Angelini, M.; Lenti, S.; Santucci, G. (2017). \"CRUMBS: A cyber security framework browser\". Proceedings from the 2017 IEEE Symposium on Visualization for Cyber Security: 1\u20138. doi:10.1109\/VIZSEC.2017.8062194.   \n\n\u2191 Abrams, M.; Weiss, J. (23 July 2008). \"Malicious Control System Cyber Security Attack Case Study\u2013 Maroochy Water Services, Australia\" (PDF). MITRE Corporation. https:\/\/www.mitre.org\/sites\/default\/files\/pdf\/08_1145.pdf . Retrieved 29 January 2018 .   \n\n\u2191 Casey, T.; Fiftal, K.; Landfield, K. et al. (January 2015). \"The Cybersecurity Framework in Action: An Intel Use Case\" (PDF). Intel Corporation. https:\/\/supplier.intel.com\/static\/governance\/documents\/The-cybersecurity-framework-in-action-an-intel-use-case-brief.pdf . Retrieved 30 January 2018 .   \n\n\u2191 Biological Sciences Division (April 2016). \"Applying\tthe Cybersecurity Framework at the University of Chicago \u2013 An Education Case Study\" (PDF). University of Chicago. http:\/\/security.bsd.uchicago.edu\/wp-content\/uploads\/sites\/2\/2016\/04\/BSD-Framework-Implementation-Case-Study_final_edition.pdf . Retrieved 31 January 2018 .   \n\n\u2191 Sweeney, S., Young, L.R. (October 2015). \"How the University of Pittsburgh Is Using the NIST Cybersecurity Framework\". Carnegie Mellon University. https:\/\/resources.sei.cmu.edu\/library\/asset-view.cfm?assetid=445056 . Retrieved 01 february 2018 .   \n\n\u2191 Montesino, R.; Fenz, S.; Baluja, W. (2012). \"SIEM\u2010based framework for security controls automation\". Information Management & Computer Security 20 (4): 248\u201363. doi:10.1108\/09685221211267639.   \n\n\u2191 Kim, E.B. (2014). \"Recommendations for information security awareness training for college students\". Information Management & Computer Security 22 (1): 115-126. doi:10.1108\/IMCS-01-2013-0005.   \n\n\u2191 BSI (July 2011). \"Embedding world-class information security management as the platform for rapid business growth\" (PDF). https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Thames-Security-UK-EN.pdf?epslanguage=en-MY . Retrieved 15 February 2018 .   \n\n\u2191 BSI (October 2012). \"How Fredrickson has reduced third party scrutiny and protected its reputation with ISO 27001 certification\" (PDF). https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Fredrickson-International-EN-UK.pdf?epslanguage=en-MY . Retrieved 15 February 2018 .   \n\n\u2191 BSI (April 2013). \"Implementing best practice and improving client confidence with ISO\/IEC 27001\" (PDF). https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Legal-Ombudsman-UK-EN.pdf . Retrieved 15 February 2018 .   \n\n\u2191 BSI (June 2013). \"Supporting business growth with ISO\/IEC 27001\" (PDF). https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-SVM-UK-EN.pdf . Retrieved 15 February 2018 .   \n\n\u2191 BSI (2013). \"InfoView Case Study\" (PDF). Archived from Studies\/BSI Infoview Case Study.pdf the original on 24 April 2016. https:\/\/web.archive.org\/web\/20160424133846\/https:\/\/www.bsigroup.com\/LocalFiles\/EN-AU\/_Case%20Studies\/BSI%20Infoview%20Case%20Study.pdf . Retrieved 15 February 2018 .   \n\n\u2191 BSI (May 2013). \"Using ISO\/IEC 27001 certification to increase resilience, reassure clients and gain a competitive edge\" (PDF). https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Capgemini-UK-EN.pdf . Retrieved 15 February 2018 .   \n\n\u2191 BSI (June 2013). \"Integrating management systems to improve business performance and achieve sustained competitive advantage\" (PDF). https:\/\/www.bsigroup.com\/Documents\/iso-22301\/case-studies\/Costain-case-study-UK-EN.pdf . Retrieved 15 February 2018 .   \n\n\u2191 Committee of Sponsoring Organizations of the Treadway Commission. \"Guidance on Enterprise Risk Management\". https:\/\/www.coso.org\/Pages\/erm.aspx . Retrieved 06 March 2018 .   \n\n\u2191 Center for Internet Security. \"CIS Controls\". https:\/\/www.cisecurity.org\/controls\/ . Retrieved 06 March 2018 .   \n\n\u2191 Information Security Forum. \"The ISF Standard of Good Practice for Information Security 2018\". https:\/\/www.securityforum.org\/tool\/the-isf-standard-good-practice-information-security-2018\/ . Retrieved 08 March 2018 .   \n\n\u2191 Brookson, C. (September 2017). \"Cybersecurity: Overview of Cybersecurity\". ETSI. https:\/\/www.enisa.europa.eu\/events\/enisa-cscg-2017\/presentations\/brookson . Retrieved 07 March 2018 .   \n\n\u2191 SABSA. \"SENC: SABSA Enhanced NIST Cybersecurity Framework\". https:\/\/sabsa.org\/sabsa-nist-framework-project\/ . Retrieved 21 March 2018 .   \n\n\u2191 IASME. \"About Cyber Essentials\". https:\/\/iasme.co.uk\/cyberessentials\/about-cyber-essentials\/ . Retrieved 07 March 2018 .   \n\n\u2191 Fraser, B., ed. (September 1997). \"RFC 2196 - Site Security Handbook\". IETF. https:\/\/www.ietf.org\/rfc\/rfc2196.txt . Retrieved 08 March 2018 .   \n\n\u2191 HITRUST (September 2017). \"Introduction to the HITRUST CSF, Version 9.0\" (PDF). https:\/\/hitrustalliance.net\/documents\/csf_rmf_related\/v9\/CSFv9Introduction.pdf . Retrieved 21 March 2018 .   \n\n\u2191 Vinson & Elkins (2014). \"Summary of CIP Version 5 Standards\" (PDF). https:\/\/www.velaw.com\/uploadedfiles\/vesite\/resources\/summarycipversion5standards2014.pdf . Retrieved 12 February 2018 .   \n\n\u2191 OSA. \"OSA Landscape\". http:\/\/www.opensecurityarchitecture.org\/cms\/foundations\/osa-landscape . Retrieved 15 March 2018 .   \n\n\u2191 CYSEC. \"GPG13 Executive Summary\". http:\/\/gpg13.com\/executive-summary\/ . Retrieved 13 March 2018 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance. Some original references had broken URLs; this version updates them to functional URLs. In one case, an archived version of the article had to be used.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\">https:\/\/www.limswiki.org\/index.php\/Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (with rendered math)LIMSwiki journal articles (all)LIMSwiki journal articles on cybersecurity\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 27 January 2020, at 23:36.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 559 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","5ed990f0be9ad0fbc7220eb00577224e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_A_security_review_of_local_government_using_NIST_CSF_A_case_study skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:A security review of local government using NIST CSF: A case study<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Evaluating <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/Risk_assessment\" title=\"Risk assessment\" class=\"wiki-link\" data-key=\"e06196a3e3b70b62af61a54e93579f37\">risk<\/a> is a challenging task regardless of an organization\u2019s nature of business or size, yet it remains an essential activity. This paper uses the National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF) to assess the cybersecurity posture of a local government organization in Western Australia. Our approach enabled the quantification of risks for specific NIST CSF core functions and respective categories and allowed making recommendations to address the gaps discovered to attain the desired level of compliance. This has led the organization to strategically target areas related to their people, processes, and technologies, thus mitigating current and future threats.\n<\/p><p><b>Keywords<\/b>: NIST Cybersecurity Framework, local government, cybersecurity, risk assessment \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>The National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF)<sup id=\"rdp-ebb-cite_ref-NISTCyber_1-0\" class=\"reference\"><a href=\"#cite_note-NISTCyber-1\">[1]<\/a><\/sup> is a risk-based approach to <a href=\"https:\/\/www.limswiki.org\/index.php\/Risk_management\" title=\"Risk management\" class=\"wiki-link\" data-key=\"ecfbfb2550c23f053bd1aa0d9ea63e4e\">manage risks<\/a> organizations face from a cybersecurity perspective. Similarly, several frameworks such as NIST SP 800-53<sup id=\"rdp-ebb-cite_ref-NISTAss14_2-0\" class=\"reference\"><a href=\"#cite_note-NISTAss14-2\">[2]<\/a><\/sup>, COBIT5<sup id=\"rdp-ebb-cite_ref-ISACA_COBIT5_3-0\" class=\"reference\"><a href=\"#cite_note-ISACA_COBIT5-3\">[3]<\/a><\/sup>, <a href=\"https:\/\/www.limswiki.org\/index.php\/ISO\/IEC_27000-series\" title=\"ISO\/IEC 27000-series\" class=\"wiki-link\" data-key=\"3fa8bbc4a9a47710224a119cf4035d96\">ISO\/IEC 27001:2013<\/a><sup id=\"rdp-ebb-cite_ref-ISO27001_4-0\" class=\"reference\"><a href=\"#cite_note-ISO27001-4\">[4]<\/a><\/sup>, ISA 62443-2-1:2009<sup id=\"rdp-ebb-cite_ref-ISA62443-2-1_5-0\" class=\"reference\"><a href=\"#cite_note-ISA62443-2-1-5\">[5]<\/a><\/sup>, and ISA 62443-3-3:2013<sup id=\"rdp-ebb-cite_ref-ISA62443-3-3_6-0\" class=\"reference\"><a href=\"#cite_note-ISA62443-3-3-6\">[6]<\/a><\/sup> are being used to <a href=\"https:\/\/www.limswiki.org\/index.php\/Risk_assessment\" title=\"Risk assessment\" class=\"wiki-link\" data-key=\"e06196a3e3b70b62af61a54e93579f37\">assess<\/a> cybersecurity risk from different perspectives, and outcomes are measured using different yardsticks. Often, navigating the various frameworks can be challenging for organizations, especially if such expertise are not present internally. Given the rapidly changing technology and threat landscape, assessing the cybersecurity posture of an organization, regardless of their business or size, is paramount.\n<\/p><p>The main goals of this paper are:\n<\/p>\n<ul><li> detailing the adoption of the NIST CSF as an assessment tool that targets different levels of the organization, depending on their level of expertise and job function to obtain responses to facilitate assessment;<\/li>\n<li> quantifying the assessment to reflect severity of actual risk, which in turn enables the organization to effectively address the issues to attain the desired level of compliance; and<\/li>\n<li> reviewing in detail similar frameworks used in the industry and relevant case studies.<\/li><\/ul>\n<p>The next section provides a background of the NIST CSF and its components. In tandem, we recommend the reader refer to NIST<sup id=\"rdp-ebb-cite_ref-NISTCyber_1-1\" class=\"reference\"><a href=\"#cite_note-NISTCyber-1\">[1]<\/a><\/sup> for additional details and strategies for suitable approaches to implement, which would vary from organization to organization. From there, our focus for the paper shifts to demonstrating the application of NIST CSF in a local government organization and providing recommendations based on our findings.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_NIST_CSF\">The NIST CSF<\/span><\/h2>\n<p>The NIST CSF<sup id=\"rdp-ebb-cite_ref-NISTCyber_1-2\" class=\"reference\"><a href=\"#cite_note-NISTCyber-1\">[1]<\/a><\/sup> consists of the Framework Core, the Framework Implementation Tiers, and the Framework Profiles. The Framework Core consists of five concurrent and continuous functions: identify, protect, detect, respond, and recover. We designed an assessment tool for our investigation based on these functions, which provided a systematic approach to ascertain the organizations cybersecurity risk management practices and processes.\n<\/p><p>The Framework Implementation Tiers describe the level that an organization's cybersecurity risk management practices comply with the framework. Tiers provide the context and degree to which cybersecurity risks are managed, as well as the extent to which business needs are considered in cybersecurity risk management. The assessment tool enabled the determination of the organization's Current Tier based on various internal and external factors such as their risk management practices, threat environment, legal and regulatory requirements, business\/mission objectives, and organizational constraints. Organizations should also determine the desired tier, provided it is feasible to implement, reduces cybersecurity risks, and meets organizational goals. The following are descriptions of the tier levels<sup id=\"rdp-ebb-cite_ref-NISTCyber_1-3\" class=\"reference\"><a href=\"#cite_note-NISTCyber-1\">[1]<\/a><\/sup>:\n<\/p>\n<ul><li> <i>Tier-1 (Partial)<\/i>: Risk management practices are not formalized and managed in an <i>ad hoc<\/i> manner, lack awareness of cybersecurity risks organization-wide, and do not have processes in place to collaborate with external entities.<\/li>\n<li> <i>Tier-2 (Risk Informed)<\/i>: Risk management practices are formalized but not integrated organization-wide. Cybersecurity activities are prioritized based on risks, with adequate means to perform related duties, and with informal means to communicate cybersecurity <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> internally and externally.<\/li>\n<li> <i>Tier-3 (Repeatable)<\/i>: Risk management practices are formalized and policies are in-place and adaptable to cyber threats. An organization-wide approach is required to manage cybersecurity, with skilled and knowledgeable personnel required to rapidly respond, understand dependencies, and understand the role of external partners.<\/li>\n<li> <i>Tier-4 (Adaptive)<\/i>: Cybersecurity practices are based on lessons learned and predictive indicators, with continuous improvement, adaptability, and timely response. An organization-wide approach is required to manage cybersecurity risks. Cybersecurity is part of the organizational culture, and the organization actively shares with external partners.<\/li><\/ul>\n<p>The Framework Profile represents the outcomes based on the business needs the organization characterized from the Framework Core and determined using the assessment tool. Consequently, a \"current profile\" (the \u201cas is\u201d state) and a \"target profile\" (the \u201cto be\u201d state) can be used to identify opportunities for improving the cybersecurity of the organization.<sup id=\"rdp-ebb-cite_ref-NISTCyber_1-4\" class=\"reference\"><a href=\"#cite_note-NISTCyber-1\">[1]<\/a><\/sup> Framework profiles can be determined based on particular implementation scenarios, and therefore, the gap between the current profile and the target profile would vary as per scenario. In this paper, a local government-specific approach to CSF was adapted. However, industry-specific tailoring may be performed for the CSF.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methodology\">Methodology<\/span><\/h2>\n<p>The NIST CSF allowed us to design an assessment tool targeted at three levels of participants within the organization: executive, management, and technical. The rationale was to ascertain organization-wide understanding of cybersecurity risks. Hence, the assessment tool comprised of questions addressing the requirements outlined as per the NIST CSF.\n<\/p><p>The questions were selected based on the nature and relevance to the level of the participant. This is because the NIST CSF comprised of questions that were both technical and non-technical. Therefore, it would have been unrealistic to expect deep knowledge of technical operations or implementation level details from a policy level executive.\n<\/p><p>In order to assist us determine a baseline (i.e., the desired tier), additional questions were included in the assessment tool to determine the nature of the organization and its business. This was then followed by the remaining requirements comprised in the NIST CSF.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Determining_compliance\">Determining compliance<\/span><\/h3>\n<p>The compliance for each measure was based on the responses provided by the participants. They were graded as complaint, partially compliant, or non-compliant, and each was assigned scores of either 10, 5, or 0, respectively, for each core function\u2019s subcategory. Any subcategory that was not applicable based on the desired tier level was excluded from the compliance score calculation.\n<\/p><p>Given the number of security requirements for each core function\u2019s subcategory is <i>N<\/i>, then the number of applicable requirements in each subcategory given the desired tier level is <i>N\u2032<\/i>. Therefore, the total compliance score <i>C<\/i> for each core function\u2019s category can be defined as:\n<\/p><p><br \/>\n<span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/6db2d3acd722161cd83462252cb56c574cf4bd8c'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -1.338ex; width:12.594ex; height:3.843ex;\" \/><\/span>\n<\/p><p>where <i>R<\/i> is the compliance score for each category of the respective core function.\n<\/p><p><br \/>\nAdditionally, a detailed document audit was conducted on existing policies and procedures. The information technology (IT) infrastructure (internal, remote locations, and cloud) were reviewed, and a detailed internal vulnerability assessment was also conducted during our investigation.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Findings\">Findings<\/span><\/h2>\n<p>The responses provided by the executive, management, and technical participants gave insight into the organization\u2019s cybersecurity posture. Table 1 shows the summary of the compliance of NIST CSF assessment. The compliance scores were determined based on the previously presented equation.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 1.<\/b> NIST CSF compliance matrix\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Function\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Category\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Compliance (%)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Total (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"5\">Identify (ID)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Asset management (ID.AM)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"5\">36\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Business environment (ID.BE)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">75\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Governance (ID.GV)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Risk assessment (ID.RA)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Risk management strategy (ID.RM)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"6\">Protect (PR)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Access control (PR.AC)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">60\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"6\">45\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Awareness and training (PR.AT)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">70\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data security (PR.DS)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Information protection processes<br \/>and procedures (PR.IP)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Maintenance (PR.MA)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">75\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Protective technology (PR.PT)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">38\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">Detect (DE)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Anomalies and events (DE.AE)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">25\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Security continuous monitoring (DE.CM)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">43\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Governance (DE.DP)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"5\">Respond (RS)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Response planning (RS.AM)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"5\">38\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Communications (RS.CO)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Analysis (RS.AN)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mitigation (RS.MI)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Improvements (RS.IM)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">Recover (RC)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Recovery planning (RC.RP)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">100\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Improvements (RC.IM)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Communications (RC.CO)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For the \"identify\" core function, the organization scored 36%. Their ability to track assets centrally, keep management informed, and understand operational risks from a cybersecurity perspective was limited, while a strategy to manage such risks did not exist. However, the organization understood its business well and was able to set priorities to support risk management decisions.\n<\/p><p>Access to physical\/virtual assets were through authorization and well-defined processes. The staff were trained and informed adequately of information security-related duties and responsibilities. Certain aspects of data security related to confidentiality and availability were done reasonably well; however, assuring integrity of data needed improvement. Similarly, local maintenance and remote maintenance of IT infrastructure were carried out in a manner consistent to policies and procedures. However, relevant policies, processes, and procedures, as well as technology to assist the protection of information systems and relevant assets, were lacking. Therefore, in aggregate, the organization scored 45% compliance for the \"protect\" core function.\n<\/p><p>The organization scored weakest in the detection of cybersecurity incidents, with a score of 25%. Although certain monitoring activities were in place to track physical security and malicious code, timely detection of anomalous activities and detection processes were lacking or non-existent.\n<\/p><p>Despite the lack of a specific response plan to respond to cybersecurity events, the organization had measures in place to report incidents and coordinate activities to respond adequately, which resulted in a 38% compliance score for the \"respond\" core function. These practices are updated from time to time; however, mechanism to perform post-incident analysis or to mitigate future cybersecurity events has not been implemented presently.\n<\/p><p>Interestingly, the organization was well prepared to deal with recovery and resumption of core services after a cybersecurity event. The recovery plans in place are tested, updated, and improved periodically, thus receiving full compliance for the \"recover\" core functionality of the framework.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Recommendations\">Recommendations<\/span><\/h2>\n<p>Based on the findings, the following recommendations were made with respect to each core function of the NIST CSF.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Identify\">Identify<\/span><\/h3>\n<dl><dd>(a) Establish a central inventory of assets, including physical devices and systems, software, and external systems, with all required information, and prioritize based on classification, criticality, and business value.<\/dd><\/dl>\n<dl><dd>(b) Identify the organizations role in the supply chain (i.e., producer-consumer model) as it captures and retains public data, collects revenue, and provides services to its stakeholders.<\/dd><\/dl>\n<dl><dd>(c) Establish an information security policy and reference relevant federal and state policies regarding cybersecurity to ensure legal and regulatory requirements are understood and managed.<\/dd><\/dl>\n<dl><dd>(d) Identify and prioritize threats and vulnerabilities, both internal and external, to determine cybersecurity risks to the organization's operations, assets, and individuals.<\/dd><\/dl>\n<dl><dd>(e) Establish risk management processes that are managed and agreed to by stakeholders to support operational risk decisions.<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"Protect\">Protect<\/span><\/h3>\n<dl><dd>(a) Strengthen the access control policy and procedures for organization-wide assets that require both physical and remote access.<\/dd><\/dl>\n<dl><dd>(b) Sensitize and increase awareness about cybersecurity throughout the workforce more comprehensively, and provide adequate cybersecurity training based on roles and responsibilities. In this regard, clearly describe cybersecurity roles and responsibilities for relevant staff and external stakeholders.<\/dd><\/dl>\n<dl><dd>(c) Enforce required provisions for data security in the policy, and implement data-at-rest and data-in-transit security, as well as integrity-checking mechanisms to ensure confidentiality, integrity, and availability of information and data.<\/dd><\/dl>\n<dl><dd>(d) Establish required policies, processes, and procedures to manage protection of information assets. This include the establishment of lacking policies and processes, particularly for configuration management, data destruction, and physical operating environment; identification of security baselines; SDLC for system management; and formulation of vulnerability, response, and recovery plans.<\/dd><\/dl>\n<dl><dd>(e) Strengthen processes that control and log remote access to organizational assets by external maintenance contractors.<\/dd><\/dl>\n<dl><dd>(f) Establish a central log of organization-wide information systems and devices, establish removable media policy, and strengthen network segregation to protect communication and control networks.<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"Detect\">Detect<\/span><\/h3>\n<dl><dd>(a) Determine baselines for network operations and data flows, and implement appropriate activities to detect and analyze events based on event data aggregated from multiple sources and sensors. Determine incident impact and threshold to prepare and allocate resources appropriately.<\/dd><\/dl>\n<dl><dd>(b) Implement tools to monitor cyber and physical environments to detect unauthorized mobile code, external service provider activities, and unauthorized access. Perform organization-wide vulnerabilities regularly.<\/dd><\/dl>\n<dl><dd>(c) Outline detection requirements in information security policy, and continuously improve these processes to ensure timely and adequate awareness of anomalous events.<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"Respond\">Respond<\/span><\/h3>\n<dl><dd>(a) Establish processes and procedures to respond to cybersecurity events in a timely manner.<\/dd><\/dl>\n<dl><dd>(b) Define cybersecurity roles and responsibilities in information security policy to ensure activities are coordinated for internal and external stakeholders, including law enforcement, in response to cybersecurity events.<\/dd><\/dl>\n<dl><dd>(c) Implement required cybersecurity event notification and detection systems to ensure adequate information is available to analyze and understand the impact to support recovery activities.<\/dd><\/dl>\n<dl><dd>(d) Implement required cybersecurity controls to detect, report, and contain incidents to prevent escalation of an incident, mitigate its effect, and eradicate the incidents.<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"Discussing_the_recommendations\">Discussing the recommendations<\/span><\/h3>\n<p>Each of the above recommendations also had specific internal stakeholder(s) identified to indicate ownership and responsibility for addressing the issues associated. Consequently, the organization was then able to develop strategies to address the issues identified and assign specific tasks to individuals. For this purpose, the organization established an internal document using Microsoft Power BI<sup id=\"rdp-ebb-cite_ref-MicrosoftPowerBI_7-0\" class=\"reference\"><a href=\"#cite_note-MicrosoftPowerBI-7\">[7]<\/a><\/sup> (typically referred to as a Power BI site) to track and visualize the status of the NIST CSF assessment (Fig. 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Ibrahim_JoSupercomp2018_74-10.png\" class=\"image wiki-link\" data-key=\"e3f43c5e3c06fa9bfc6bb92a45a790f7\"><img alt=\"Fig1 Ibrahim JoSupercomp2018 74-10.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/13\/Fig1_Ibrahim_JoSupercomp2018_74-10.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Microsoft Power BI Internal Site for tracking, visualizing, and reporting NIST CSF assessment findings, courtesy of the participating local government organization<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The Power BI site facilitated transparency, visibility, and central reporting throughout the organization. Intuitively, this resulted in a rapid and responsive drive for the organization to address and prioritize issues based on severity and cost, with the goal of achieving Tier-2 compliance.\n<\/p><p>Furthermore, a desire to achieve a higher compliance level such as Tier-3 was expressed. Such aspiration is encouraged, however, with caution. Even though a higher level of compliance will improve the cybersecurity posture of the organization, it will also affect other aspects such as resources and cost. For example, contrast the risk management process between Tier-2 and Tier-3 as defined in the NIST CSF<sup id=\"rdp-ebb-cite_ref-NISTCyber_1-5\" class=\"reference\"><a href=\"#cite_note-NISTCyber-1\">[1]<\/a><\/sup>:\n<\/p>\n<dl><dd>(a) Implementation of risk management practices are not mandatory in Tier-2, whereas these have to be implemented as organization-wide policies in Tier-3. Thus, Tier-3 organizations should have the procedures, processes, technology, and human resources to implement relevant policies.<\/dd><\/dl>\n<dl><dd>(b) The cybersecurity activities\u2019 priorities are updated in a passive nature in Tier-2 as opposed to regular active updates and constant re-evaluation of priorities for Tier-3 compliance. To acquire such capability, an organization requires adequate technology, skilled human resources, and relevant policies that would enable keeping pace with the changes in the technology and threat landscape.<\/dd><\/dl>\n<p>In addition to the two points highlighted above, considering both the integrated risk management program and external participation<sup id=\"rdp-ebb-cite_ref-NISTCyber_1-6\" class=\"reference\"><a href=\"#cite_note-NISTCyber-1\">[1]<\/a><\/sup>, significant investment in resources and human skills development or acquisition is needed to make the transition from Tier-2 to Tier-3. Moreover, this should only be considered carefully based on the organization\u2019s business requirements, strategic objectives, budget, risk appetite, and current and future threats.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Related_frameworks\">Related frameworks<\/span><\/h2>\n<p>The diversity and complexity of information technology (IT) system components have increased significantly in recent years. Consequently, in order for businesses to adequately secure these systems, several standards and frameworks have been developed.<sup id=\"rdp-ebb-cite_ref-AngeliniCRUMBS17_8-0\" class=\"reference\"><a href=\"#cite_note-AngeliniCRUMBS17-8\">[8]<\/a><\/sup> Such frameworks need to be applicable to all manner of business sectors, be they government or private, enterprise or small-business.\n<\/p><p>Since NIST CSF can be considered a high-level abstraction of related frameworks, it provides references to other related frameworks for specific implementation guidelines. These referenced frameworks include:\n<\/p>\n<ul><li> NIST SP 800-53 Rev. 4<\/li>\n<li> Control Objectives for Information and Related Technologies (COBIT5)<\/li>\n<li> ISO\/IEC 27001:2013<\/li>\n<li> ISA 62443-2-1:2009<\/li>\n<li> ISA 62443-3-3:2013<\/li><\/ul>\n<p>These are further described below.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"NIST_SP_800-53_Rev._4\">NIST SP 800-53 Rev. 4<\/span><\/h3>\n<p>NIST SP 800-53<sup id=\"rdp-ebb-cite_ref-NISTAss14_2-1\" class=\"reference\"><a href=\"#cite_note-NISTAss14-2\">[2]<\/a><\/sup> revisions are made according to changes in responsibilities under the Federal Information Security Management Act (FISMA), Public Law (P.L.) 107-347. Revision 4 of this framework consists of five functions (identify, protect, detect, respond, and recover), 22 categories, and 98 subcategories. This framework utilizes a four-tier security model (partial, risk informed, repeatable, and adaptive) and a seven-step process (prioritize and scope, orient, create a current profile, conduct a risk assessment, create a target profile, determine, analyze and prioritize gaps, and implement action plan). It focuses on assessing the current situation by determining how to assess security, how to consider risk, and how to resolve the security threats.\n<\/p><p>Table 2 provides a summary of useful examples of how the NIST SP 800-53 framework has been applied in practice.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 2.<\/b> Summary of case studies for NIST SP 800-53\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Case study\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Description\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Maroochy water services cyber attack against critical infrastructure in 2000<sup id=\"rdp-ebb-cite_ref-AbramsMalicious08_9-0\" class=\"reference\"><a href=\"#cite_note-AbramsMalicious08-9\">[9]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Disgruntled former employee used insider knowledge and stolen configurations and equipment to release more than one million liters of untreated sewerage water resulting in considerable environmental damage and prosecution by the Environmental Protection Agency. The case study revealed that the application of CSF controls would have mitigated the cyber attack.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Intel\u2019s high-risk IT business units\u2019 pilot project<sup id=\"rdp-ebb-cite_ref-CaseyTheCyber15_10-0\" class=\"reference\"><a href=\"#cite_note-CaseyTheCyber15-10\">[10]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Intel IT\u2019s Office and Enterprise business units, considered to be high-level risk environments, were the test bed for a pilot project to test the effectiveness of the NIST SP 800-53. The benefits of using this framework were realized within a short time-frame, with coherent use of risk management technologies across the business model, improved identification of strengths and weaknesses, and more efficient assessments of security priorities. As a result of the pilot project, Intel IT planned to expand the framework\u2019s implementation throughout their business infrastructure.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cybersecurity framework implementation at the University of Chicago<sup id=\"rdp-ebb-cite_ref-BSDApply16_11-0\" class=\"reference\"><a href=\"#cite_note-BSDApply16-11\">[11]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The University of Chicago used the framework to establish cybersecurity protection for its Biological Services Division (BSD). The four-part implementation consisted of identifying the initial state of cybersecurity processes, assessment of the initial threat landscape, determination of the desired target status, and creation of a roadmap to establish and monitor progress. This resulted in improved identification of security requirements and target objectives, better development and maintenance of departmental processes to achieve these objectives, more long-term security solutions in a cost-effective manner, and improved information-sharing and good work practices across departments with different cybersecurity requirements.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">How the University of Pittsburgh is using the NIST Cybersecurity Framework<sup id=\"rdp-ebb-cite_ref-SweeneyHowThe15_12-0\" class=\"reference\"><a href=\"#cite_note-SweeneyHowThe15-12\">[12]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The University of Pittsburgh used NIST SP 800-53 to implement an IT security package that would cater for diversified needs while enabling collaboration between different departments, accommodate a wide variety of information types and sensitivities, and encompass third-party contractors on an <i>ad hoc<\/i> basis. NIST SP 800-53 enabled these goals to be met through the streamlining of existing practices and improving documentation. The scalable nature of NIST CSF was applicable to the differing scope and IT requirements of each department within the University.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SIEM-based framework for security controls automation<sup id=\"rdp-ebb-cite_ref-MontesinoSIEM12_13-0\" class=\"reference\"><a href=\"#cite_note-MontesinoSIEM12-13\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The potential of using SIEM technology is investigated with the aim of maximizing security-control automation. For the security controls identified in NIST SP 800-53, approximately 30% of these controls were considered as having the capability of automation control. The cost of implementing a SIEM-based framework for security-control automation would be quickly recouped within a short time compared to the reduced employee hours required to monitor an infrastructure the size of a local government organization.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Recommendations for information security awareness training for college students<sup id=\"rdp-ebb-cite_ref-KimRecomm14_14-0\" class=\"reference\"><a href=\"#cite_note-KimRecomm14-14\">[14]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A survey largely based on NIST SP 800-50 was designed to assess information security awareness among students at the business college of a mid-sized university in New England. The survey found that less than one-quarter of the participants had undertaken any form of information security awareness training (ISAT), and only two of the 68 had enrolled in university-provided training. ISAT of employees in local government is an integral part of a well-implemented cybersecurity infrastructure. Any cybersecurity review needs to ascertain current levels of information security awareness to gauge whether existing training is effective or deficient. The training needs to be regularly updated as new vulnerabilities and threats continually develop in this field.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Control_Objectives_for_Information_and_Related_Technologies_.28COBIT5.29\">Control Objectives for Information and Related Technologies (COBIT5)<\/span><\/h3>\n<p>COBIT5<sup id=\"rdp-ebb-cite_ref-ISACA_COBIT5_3-1\" class=\"reference\"><a href=\"#cite_note-ISACA_COBIT5-3\">[3]<\/a><\/sup> is a business CSF designed for the governance and maintenance of enterprise IT systems. It consists of five domains and 37 processes in-line with the responsibility areas of plan, build, run, and monitor. COBIT5 is aligned and coordinated with other recognized IT standards and good practices, such as NIST, ISO 27000, COSO, ITIL, BiSL, CMMI, TOGAF, and PMBOK. It is built around the following considerations:\n<\/p>\n<ul><li> meeting stakeholder expectations.<\/li>\n<li> controlling enterprise process end-to-end<\/li>\n<li> working as a single integrated framework<\/li>\n<li> recognizing that \u201cmanagement\u201d and \u201cgovernance\u201d are two different things<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"ISO.2FIEC_27001:2013\">ISO\/IEC 27001:2013<\/span><\/h3>\n<p>ISO\/IEC 27001:2013<sup id=\"rdp-ebb-cite_ref-ISO27001_4-1\" class=\"reference\"><a href=\"#cite_note-ISO27001-4\">[4]<\/a><\/sup> is an international information security standard published by the <a href=\"https:\/\/www.limswiki.org\/index.php\/International_Organization_for_Standardization\" title=\"International Organization for Standardization\" class=\"wiki-link\" data-key=\"116defc5d89c8a55f5b7c1be0790b442\">International Organization for Standardization<\/a> (ISO) and the International Electrotechnical Commission (IEC), which originated from the British Standard, BS 7799. This framework consists of 114 controls in 14 groups describing the requirements needed to design and implement an information security management system (ISMS). Version 2, released in 2013, replaces the original 2005 version. It is a standard that should be implemented by all businesses where information security is a critical factor, but in particular, applies to software development, managed service providers\/hosting services providers, IT, banking and insurance, information management, government agencies and their service providers, and e-commerce merchants.<sup id=\"rdp-ebb-cite_ref-ISO27001_4-2\" class=\"reference\"><a href=\"#cite_note-ISO27001-4\">[4]<\/a><\/sup>\n<\/p><p>Table 3 provides a summary of useful examples of how the ISO\/IEC 27001:2013 framework has been applied in practice.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 3.<\/b> Summary of case studies for ISO 27001:2013\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Case study\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Description\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Thames Security Shredding (TSS) Ltd.<sup id=\"rdp-ebb-cite_ref-BSIEmbed11_15-0\" class=\"reference\"><a href=\"#cite_note-BSIEmbed11-15\">[15]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TSS specializes in the collection and destruction of confidential documentation on a commercial scale. Maintaining information security is, therefore, a critical process to protect their clients\u2019 identity and to ensure compliance with the UK Data Protection Act 1998. Certification to the ISO\/IEC 27001 standard was seen as an integral part of the implementation and maintenance of world-class customer-centric security controls that would satisfy existing and prospective customers\u2019 needs and allow for rapid growth in the business. ISO\/IEC 27001 certification resulted in an improved attitude and awareness of their staff towards information-security-related issues. A risk-based business continuity plan was used to minimize the impact of any potential security breaches. The certification allowed documentation to be continually updated and improved as corrective actions were taken.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fredrickson International<sup id=\"rdp-ebb-cite_ref-BSIHowFred12_16-0\" class=\"reference\"><a href=\"#cite_note-BSIHowFred12-16\">[16]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Debt collection is a sector which\u2014like banking, finance, telecommunications, and local government\u2014is coming under increasing scrutiny and regulation. Fredrickson International is a debt collection agency that lists a central government department, as well as several U.K. financial institutions and FTSE 100 companies, among their clients. Since attaining ISO\/IEC 27001 certification, Fredrickson has achieved higher levels of security awareness throughout its departments, staff, and employees. Security audits have become more streamlined, and customers were given the confidence that Fredrickson was conducting international best practice when it came to information security.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Legal Ombudsman<sup id=\"rdp-ebb-cite_ref-BSImplem13_17-0\" class=\"reference\"><a href=\"#cite_note-BSImplem13-17\">[17]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The office of the Legal Ombudsman for England and Wales was established to simplify the process by which members of the public, small businesses, charities, clubs, trusts, etc. could resolve complaints about legal practitioners. To improve its customer service, information security practices conducted by the office needed to show that greater information security awareness had been established, diligence and compliance in handling sensitive information were in place, and that an assurance framework was aligned with global best practice. ISO\/IEC 27001 certification helped the Legal Ombudsman for England and Wales to provide clients with the confidence and reassurance that it was conducting its work by the highest work standards. A better understanding of the information security among its staff led to a reduction in risk and an increase in productivity.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SVM Cards Europe<sup id=\"rdp-ebb-cite_ref-BSISupport13_18-0\" class=\"reference\"><a href=\"#cite_note-BSISupport13-18\">[18]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SVM is a leading provider of gift card, voucher, e-code, reward code, and similar promotional and benefit schemes throughout Europe. SVM required secure business processes, improved internal organization, and increased information protection. It also sought greater tender and competitive advantage. With ISO\/IEC 27001 certification, SVM observed that processes became more of a lifestyle than strictly about security only, which resulted in less downtime, instigated a stronger organizational structure, improved on its ability to win new contracts, and have greater confidence that their information security processes were working properly.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">InfoView Technologies<sup id=\"rdp-ebb-cite_ref-BSIInfoview13_19-0\" class=\"reference\"><a href=\"#cite_note-BSIInfoview13-19\">[19]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">InfoView Technologies, a Queensland-based data analytics company, required a business model that met state government requirements, improved data security understanding, became more professional, improved its business culture, and was able to sustain and continuously improve its information security management, systems, and policies. These goals were achieved through ISO\/IEC 27001 accreditation, after which InfoView Technologies were able to gain increased market access, meet compliance requirements of the Queensland state government, reduce risk, become more competitive, and streamline its practices and business culture.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Capgemini<sup id=\"rdp-ebb-cite_ref-BSIUsingISO13_20-0\" class=\"reference\"><a href=\"#cite_note-BSIUsingISO13-20\">[20]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Capgemini is the largest IT services company in Europe and a global leader in its multiple domains of services. Operating in more than 40 countries, and over 100 languages, Capgemini\u2019s business model needed to transcend national and cultural boundaries. Systems were required to be robust to avoid losing business and maintain competitiveness. Protection of client assets and resources was deemed a priority to assure confidentiality, integrity, and availability. Through ISO\/IEC 27001 certification, Capgemini was able to ensure improved security within its departments and for its clients, enhance security awareness in its staff and employees, and provide more efficient and streamlined documentation and reporting procedures. Standards certification needed to be applicable within the global marketplace and remain pertinent regardless of cultural differences.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Costain<sup id=\"rdp-ebb-cite_ref-BSIIntegrating13_21-0\" class=\"reference\"><a href=\"#cite_note-BSIIntegrating13-21\">[21]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Costain, a U.K.-based engineering and construction group, has contributed to the construction of significant projects worldwide. Obtaining standards certifications was seen as the correct path to achieve improvements in several internal processes. Such goals required the implementation of several standards, such as quality management standard (ISO 9001), environmental management (ISO 14001), health and safety (BS OHSAS 18001), collaborative business relationships (BS 11000), information security management (ISO\/IEC 27001), and business continuity management (ISO 22301). Through the enactment of multiple standards, Costain was able to improve several areas of their business to the benefit of their internal and external customers.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"ISA_62443-2-1:2009\">ISA 62443-2-1:2009<\/span><\/h3>\n<p>ISA 62443-2-1:2009<sup id=\"rdp-ebb-cite_ref-ISA62443-2-1_5-1\" class=\"reference\"><a href=\"#cite_note-ISA62443-2-1-5\">[5]<\/a><\/sup> is an International Standards on Auditing (ISA) standard covering the elements required to develop an industrial automation control system - security management system (IACS-SMS). It consists of three categories, three element groups, and 22 elements. The framework is the first of four ISA policy and procedure products that identifies the essentials necessary to establish an effective cybersecurity management system (CSMS). However, the step-by-step approach as to how this is achieved is company-specific and according to their own business culture. These essentials addressed by this standard include:\n<\/p>\n<ul><li> analyzing risk<\/li>\n<li> addressing risk with the CSMS<\/li>\n<li> monitoring and improving the CSMS<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"ISA_62443-3-3:2013\">ISA 62443-3-3:2013<\/span><\/h3>\n<p>ISA 62443-3-3:2013<sup id=\"rdp-ebb-cite_ref-ISA62443-3-3_6-1\" class=\"reference\"><a href=\"#cite_note-ISA62443-3-3-6\">[6]<\/a><\/sup> is an International Standards on Auditing (ISA) standard covering the elements required for cybersecurity controls of industrial control systems (ICS). It consists of seven foundation requirements and 51 system requirements.\n<\/p><p>ISA 62443-3-3:2013 is the third of three ISA systems products that outlines system security requirements and security levels.<sup id=\"rdp-ebb-cite_ref-ISA62443-3-3_6-2\" class=\"reference\"><a href=\"#cite_note-ISA62443-3-3-6\">[6]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Other_frameworks\">Other frameworks<\/span><\/h3>\n<p>In addition to the above, other frameworks used in the industry include the following:\n<\/p>\n<ul><li> The Committee of Sponsoring Organizations of the Treadway Commission's (COSO) \u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200bEnterprise Risk Management standard is designed jointly by five leading associations, with the aim of integrating strategy and performance.<sup id=\"rdp-ebb-cite_ref-COSOGuidance_22-0\" class=\"reference\"><a href=\"#cite_note-COSOGuidance-22\">[22]<\/a><\/sup><\/li>\n<li> The Council on CyberSecurity (now Center for Internet Security) CIS Controls consist of a prioritized set of actions, originally developed by the SANS Institute, to protect assets from cyber attack.<sup id=\"rdp-ebb-cite_ref-CISControls_23-0\" class=\"reference\"><a href=\"#cite_note-CISControls-23\">[23]<\/a><\/sup><\/li>\n<li> ISF Standard of Good Practice for Information Security is a standard aimed at providing controls and guidance on all aspects of information security.<sup id=\"rdp-ebb-cite_ref-ISFStandard_24-0\" class=\"reference\"><a href=\"#cite_note-ISFStandard-24\">[24]<\/a><\/sup><\/li>\n<li> ETSI Cyber Security Technical Committee (TC Cyber) was developed to improve standards within the European telecommunications sector.<sup id=\"rdp-ebb-cite_ref-BrooksonOverview17_25-0\" class=\"reference\"><a href=\"#cite_note-BrooksonOverview17-25\">[25]<\/a><\/sup> <\/li>\n<li> The Sherwood Applied Business Security Architecture (SABSA) Enhanced NIST Cybersecurity (SENC) project enhances the five core levels of the NIST CSF into a SABSA model consisting of a six-level security architecture.<sup id=\"rdp-ebb-cite_ref-SABSA_SENC_26-0\" class=\"reference\"><a href=\"#cite_note-SABSA_SENC-26\">[26]<\/a><\/sup><\/li>\n<li> The IASME Consortium's Cyber Essentials is an information assurance standard based on ISO 27000 but aimed at small businesses.<sup id=\"rdp-ebb-cite_ref-IASMEAboutCyber_27-0\" class=\"reference\"><a href=\"#cite_note-IASMEAboutCyber-27\">[27]<\/a><\/sup><\/li>\n<li> The IETF's <a class=\"external mw-magiclink-rfc\" rel=\"external_link\" href=\"https:\/\/tools.ietf.org\/html\/rfc2196\" target=\"_blank\">RFC 2196<\/a> Site Security Handbook represents a guide on how to develop computer security policies and procedures.<sup id=\"rdp-ebb-cite_ref-FraserSite97_28-0\" class=\"reference\"><a href=\"#cite_note-FraserSite97-28\">[28]<\/a><\/sup><\/li>\n<li> Health Information Trust Alliance's HITRUST CSF is the first IT security CSF designed specifically for the healthcare sector. It is based on existing NIST standards and is aimed at healthcare and information security professionals.<sup id=\"rdp-ebb-cite_ref-HITRUSTIntro17_29-0\" class=\"reference\"><a href=\"#cite_note-HITRUSTIntro17-29\">[29]<\/a><\/sup><\/li>\n<li> North American Electric Reliability Corporation Critical Infrastructure Protection (NERC-CIP), Version 5 is a set of requirements needed to secure the assets of the North American bulk electric system.<sup id=\"rdp-ebb-cite_ref-NERCSumm14_30-0\" class=\"reference\"><a href=\"#cite_note-NERCSumm14-30\">[30]<\/a><\/sup><\/li>\n<li> Open Security Architecture (OSA) is a free community-owned resource of advice on the selection, design, and integration of devices required to provide security and control of an IT network.<sup id=\"rdp-ebb-cite_ref-OSALandscape_31-0\" class=\"reference\"><a href=\"#cite_note-OSALandscape-31\">[31]<\/a><\/sup><\/li>\n<li> Good Practice Guide 13 (GPG13) is a U.K. government CSF related to Code of Connection (CoCo) compliance for businesses to secure IT systems.<sup id=\"rdp-ebb-cite_ref-CYSEC_GPG13Exec_32-0\" class=\"reference\"><a href=\"#cite_note-CYSEC_GPG13Exec-32\">[32]<\/a><\/sup><\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>In this paper we have used the NIST CSF to evaluate the cybersecurity risks of a local government organization in Western Australia. Our approach can be used to derive measurable metrics for each Framework Core function and respective categories, thus enabling the organization to ascertain the cybersecurity preparedness to actual risk.\n<\/p><p>Our findings suggest that evaluating the desired tier compliance to the NIST CSF helps identify the specific people, processes, and technology areas that require improvement (i.e., gaps), which directly influence threat mitigation. The application of CSF helped us understand the current security context of the organization while identifying the risks and future growth areas to improve. While higher tier compliance may be desired, we have also recommended that the organization\u2019s business requirements, strategic goals, budget, risk appetite, and current and future threats be considered carefully.\n<\/p><p>Furthermore, as we have presented several related frameworks, navigating such frameworks for self-assessment can be challenging\u2014often not intended by design\u2014but not impossible. We have observed that the NIST CSF offers an advantage over other frameworks in this regard. However, there is still room for developing additional tools that would simplify the implementation process and speed up adoption.\n<\/p><p>Therefore, our future work will aim to improve the current assessment tool we have used, with a focus of making it adaptable and accessible to a wider audience and measurable for accurate quantification of cyber preparedness.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We would like to thank the Western Australia local government organization for sharing their case study for this research. We would also like to thank their staff for their support and cooperation during the assessment.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-NISTCyber-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NISTCyber_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-NISTCyber_1-1\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-NISTCyber_1-2\">1.2<\/a><\/sup> <sup><a href=\"#cite_ref-NISTCyber_1-3\">1.3<\/a><\/sup> <sup><a href=\"#cite_ref-NISTCyber_1-4\">1.4<\/a><\/sup> <sup><a href=\"#cite_ref-NISTCyber_1-5\">1.5<\/a><\/sup> <sup><a href=\"#cite_ref-NISTCyber_1-6\">1.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Institute of Standards and Technology. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nist.gov\/cyberframework\" target=\"_blank\">\"Cybersecurity Framework\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nist.gov\/cyberframework\" target=\"_blank\">https:\/\/www.nist.gov\/cyberframework<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cybersecurity+Framework&rft.atitle=&rft.aulast=National+Institute+of+Standards+and+Technology&rft.au=National+Institute+of+Standards+and+Technology&rft_id=https%3A%2F%2Fwww.nist.gov%2Fcyberframework&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NISTAss14-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NISTAss14_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-NISTAss14_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Institute of Standards and Technology (December 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/nvlpubs.nist.gov\/nistpubs\/SpecialPublications\/NIST.SP.800-53Ar4.pdf\" target=\"_blank\">\"Assessing Security and Privacy Controls in Federal Information Systems and Organizations\"<\/a> (PDF). <i>NIST Special Publication 800-53A, Revision 4<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/nvlpubs.nist.gov\/nistpubs\/SpecialPublications\/NIST.SP.800-53Ar4.pdf\" target=\"_blank\">https:\/\/nvlpubs.nist.gov\/nistpubs\/SpecialPublications\/NIST.SP.800-53Ar4.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Assessing+Security+and+Privacy+Controls+in+Federal+Information+Systems+and+Organizations&rft.atitle=NIST+Special+Publication+800-53A%2C+Revision+4&rft.aulast=National+Institute+of+Standards+and+Technology&rft.au=National+Institute+of+Standards+and+Technology&rft.date=December+2014&rft_id=https%3A%2F%2Fnvlpubs.nist.gov%2Fnistpubs%2FSpecialPublications%2FNIST.SP.800-53Ar4.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISACA_COBIT5-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISACA_COBIT5_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISACA_COBIT5_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">ISACA. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/cobitonline.isaca.org\/\" target=\"_blank\">\"COBIT 5\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/cobitonline.isaca.org\/\" target=\"_blank\">https:\/\/cobitonline.isaca.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=COBIT+5&rft.atitle=&rft.aulast=ISACA&rft.au=ISACA&rft_id=https%3A%2F%2Fcobitonline.isaca.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO27001-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISO27001_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISO27001_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-ISO27001_4-2\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">International Standards Organization. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/54534.html\" target=\"_blank\">\"ISO\/IEC 27001:2013 - Information technology \u2014 Security techniques \u2014 Information security management systems \u2014 Requirements\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/54534.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/54534.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO%2FIEC+27001%3A2013+-+Information+technology+%E2%80%94+Security+techniques+%E2%80%94+Information+security+management+systems+%E2%80%94+Requirements&rft.atitle=&rft.aulast=International+Standards+Organization&rft.au=International+Standards+Organization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F54534.html&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISA62443-2-1-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISA62443-2-1_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISA62443-2-1_5-1\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">ISA (13 January 2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.icsdefender.ir\/files\/scadadefender-ir\/paygahdanesh\/standards\/ISA-62443-2-1-Public.pdf\" target=\"_blank\">\"Security for industrial automation and control systems, Part 2-1: Establishing an industrial automation and control systems security program\"<\/a> (PDF). <i>ANSI\/ISA-62443-2-1 (99.02.01)-2009<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.icsdefender.ir\/files\/scadadefender-ir\/paygahdanesh\/standards\/ISA-62443-2-1-Public.pdf\" target=\"_blank\">http:\/\/www.icsdefender.ir\/files\/scadadefender-ir\/paygahdanesh\/standards\/ISA-62443-2-1-Public.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Security+for+industrial+automation+and+control+systems%2C+Part+2-1%3A+Establishing+an+industrial+automation+and+control+systems+security+program&rft.atitle=ANSI%2FISA-62443-2-1+%2899.02.01%29-2009&rft.aulast=ISA&rft.au=ISA&rft.date=13+January+2009&rft_id=http%3A%2F%2Fwww.icsdefender.ir%2Ffiles%2Fscadadefender-ir%2Fpaygahdanesh%2Fstandards%2FISA-62443-2-1-Public.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISA62443-3-3-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISA62443-3-3_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISA62443-3-3_6-1\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-ISA62443-3-3_6-2\">6.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">ISA (12 August 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.icsdefender.ir\/files\/scadadefender-ir\/paygahdanesh\/standards\/ISA-62443-3-3-Public.pdf\" target=\"_blank\">\"Security for industrial automation and control systems, Part 3-3: System security requirements and security levels\"<\/a> (PDF). <i>ANSI\/ISA-62443-3-3 (99.03.03)-2013<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.icsdefender.ir\/files\/scadadefender-ir\/paygahdanesh\/standards\/ISA-62443-3-3-Public.pdf\" target=\"_blank\">http:\/\/www.icsdefender.ir\/files\/scadadefender-ir\/paygahdanesh\/standards\/ISA-62443-3-3-Public.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Security+for+industrial+automation+and+control+systems%2C+Part+3-3%3A+System+security+requirements+and+security+levels&rft.atitle=ANSI%2FISA-62443-3-3+%2899.03.03%29-2013&rft.aulast=ISA&rft.au=ISA&rft.date=12+August+2013&rft_id=http%3A%2F%2Fwww.icsdefender.ir%2Ffiles%2Fscadadefender-ir%2Fpaygahdanesh%2Fstandards%2FISA-62443-3-3-Public.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MicrosoftPowerBI-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MicrosoftPowerBI_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Microsoft. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/powerbi.microsoft.com\/en-us\/\" target=\"_blank\">\"Microsoft Power BI\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/powerbi.microsoft.com\/en-us\/\" target=\"_blank\">https:\/\/powerbi.microsoft.com\/en-us\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 April 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Microsoft+Power+BI&rft.atitle=&rft.aulast=Microsoft&rft.au=Microsoft&rft_id=https%3A%2F%2Fpowerbi.microsoft.com%2Fen-us%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AngeliniCRUMBS17-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AngeliniCRUMBS17_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Angelini, M.; Lenti, S.; Santucci, G. (2017). \"CRUMBS: A cyber security framework browser\". <i>Proceedings from the 2017 IEEE Symposium on Visualization for Cyber Security<\/i>: 1\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FVIZSEC.2017.8062194\" target=\"_blank\">10.1109\/VIZSEC.2017.8062194<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=CRUMBS%3A+A+cyber+security+framework+browser&rft.jtitle=Proceedings+from+the+2017+IEEE+Symposium+on+Visualization+for+Cyber+Security&rft.aulast=Angelini%2C+M.%3B+Lenti%2C+S.%3B+Santucci%2C+G.&rft.au=Angelini%2C+M.%3B+Lenti%2C+S.%3B+Santucci%2C+G.&rft.date=2017&rft.pages=1%E2%80%938&rft_id=info:doi\/10.1109%2FVIZSEC.2017.8062194&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbramsMalicious08-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbramsMalicious08_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Abrams, M.; Weiss, J. (23 July 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.mitre.org\/sites\/default\/files\/pdf\/08_1145.pdf\" target=\"_blank\">\"Malicious Control System Cyber Security Attack Case Study\u2013 Maroochy Water Services, Australia\"<\/a> (PDF). MITRE Corporation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.mitre.org\/sites\/default\/files\/pdf\/08_1145.pdf\" target=\"_blank\">https:\/\/www.mitre.org\/sites\/default\/files\/pdf\/08_1145.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 29 January 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Malicious+Control+System+Cyber+Security+Attack+Case+Study%E2%80%93+Maroochy+Water+Services%2C+Australia&rft.atitle=&rft.aulast=Abrams%2C+M.%3B+Weiss%2C+J.&rft.au=Abrams%2C+M.%3B+Weiss%2C+J.&rft.date=23+July+2008&rft.pub=MITRE+Corporation&rft_id=https%3A%2F%2Fwww.mitre.org%2Fsites%2Fdefault%2Ffiles%2Fpdf%2F08_1145.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyTheCyber15-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CaseyTheCyber15_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Casey, T.; Fiftal, K.; Landfield, K. et al. (January 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/supplier.intel.com\/static\/governance\/documents\/The-cybersecurity-framework-in-action-an-intel-use-case-brief.pdf\" target=\"_blank\">\"The Cybersecurity Framework in Action: An Intel Use Case\"<\/a> (PDF). Intel Corporation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/supplier.intel.com\/static\/governance\/documents\/The-cybersecurity-framework-in-action-an-intel-use-case-brief.pdf\" target=\"_blank\">https:\/\/supplier.intel.com\/static\/governance\/documents\/The-cybersecurity-framework-in-action-an-intel-use-case-brief.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Cybersecurity+Framework+in+Action%3A+An+Intel+Use+Case&rft.atitle=&rft.aulast=Casey%2C+T.%3B+Fiftal%2C+K.%3B+Landfield%2C+K.+et+al.&rft.au=Casey%2C+T.%3B+Fiftal%2C+K.%3B+Landfield%2C+K.+et+al.&rft.date=January+2015&rft.pub=Intel+Corporation&rft_id=https%3A%2F%2Fsupplier.intel.com%2Fstatic%2Fgovernance%2Fdocuments%2FThe-cybersecurity-framework-in-action-an-intel-use-case-brief.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSDApply16-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSDApply16_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Biological Sciences Division (April 2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/security.bsd.uchicago.edu\/wp-content\/uploads\/sites\/2\/2016\/04\/BSD-Framework-Implementation-Case-Study_final_edition.pdf\" target=\"_blank\">\"Applying\tthe Cybersecurity Framework at the University of Chicago \u2013 An Education Case Study\"<\/a> (PDF). University of Chicago<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/security.bsd.uchicago.edu\/wp-content\/uploads\/sites\/2\/2016\/04\/BSD-Framework-Implementation-Case-Study_final_edition.pdf\" target=\"_blank\">http:\/\/security.bsd.uchicago.edu\/wp-content\/uploads\/sites\/2\/2016\/04\/BSD-Framework-Implementation-Case-Study_final_edition.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 31 January 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Applying%09the+Cybersecurity+Framework+at+the+University+of+Chicago+%E2%80%93+An+Education+Case+Study&rft.atitle=&rft.aulast=Biological+Sciences+Division&rft.au=Biological+Sciences+Division&rft.date=April+2016&rft.pub=University+of+Chicago&rft_id=http%3A%2F%2Fsecurity.bsd.uchicago.edu%2Fwp-content%2Fuploads%2Fsites%2F2%2F2016%2F04%2FBSD-Framework-Implementation-Case-Study_final_edition.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SweeneyHowThe15-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SweeneyHowThe15_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Sweeney, S., Young, L.R. (October 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/resources.sei.cmu.edu\/library\/asset-view.cfm?assetid=445056\" target=\"_blank\">\"How the University of Pittsburgh Is Using the NIST Cybersecurity Framework\"<\/a>. Carnegie Mellon University<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/resources.sei.cmu.edu\/library\/asset-view.cfm?assetid=445056\" target=\"_blank\">https:\/\/resources.sei.cmu.edu\/library\/asset-view.cfm?assetid=445056<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 february 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+the+University+of+Pittsburgh+Is+Using+the+NIST+Cybersecurity+Framework&rft.atitle=&rft.aulast=Sweeney%2C+S.%2C+Young%2C+L.R.&rft.au=Sweeney%2C+S.%2C+Young%2C+L.R.&rft.date=October+2015&rft.pub=Carnegie+Mellon+University&rft_id=https%3A%2F%2Fresources.sei.cmu.edu%2Flibrary%2Fasset-view.cfm%3Fassetid%3D445056&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MontesinoSIEM12-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MontesinoSIEM12_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Montesino, R.; Fenz, S.; Baluja, W. (2012). \"SIEM\u2010based framework for security controls automation\". <i>Information Management & Computer Security<\/i> <b>20<\/b> (4): 248\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1108%2F09685221211267639\" target=\"_blank\">10.1108\/09685221211267639<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SIEM%E2%80%90based+framework+for+security+controls+automation&rft.jtitle=Information+Management+%26+Computer+Security&rft.aulast=Montesino%2C+R.%3B+Fenz%2C+S.%3B+Baluja%2C+W.&rft.au=Montesino%2C+R.%3B+Fenz%2C+S.%3B+Baluja%2C+W.&rft.date=2012&rft.volume=20&rft.issue=4&rft.pages=248%E2%80%9363&rft_id=info:doi\/10.1108%2F09685221211267639&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimRecomm14-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KimRecomm14_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, E.B. (2014). \"Recommendations for information security awareness training for college students\". <i>Information Management & Computer Security<\/i> <b>22<\/b> (1): 115-126. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1108%2FIMCS-01-2013-0005\" target=\"_blank\">10.1108\/IMCS-01-2013-0005<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Recommendations+for+information+security+awareness+training+for+college+students&rft.jtitle=Information+Management+%26+Computer+Security&rft.aulast=Kim%2C+E.B.&rft.au=Kim%2C+E.B.&rft.date=2014&rft.volume=22&rft.issue=1&rft.pages=115-126&rft_id=info:doi\/10.1108%2FIMCS-01-2013-0005&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSIEmbed11-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSIEmbed11_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">BSI (July 2011). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Thames-Security-UK-EN.pdf?epslanguage=en-MY\" target=\"_blank\">\"Embedding world-class information security management as the platform for rapid business growth\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Thames-Security-UK-EN.pdf?epslanguage=en-MY\" target=\"_blank\">https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Thames-Security-UK-EN.pdf?epslanguage=en-MY<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Embedding+world-class+information+security+management+as+the+platform+for+rapid+business+growth&rft.atitle=&rft.aulast=BSI&rft.au=BSI&rft.date=July+2011&rft_id=https%3A%2F%2Fwww.bsigroup.com%2FDocuments%2Fiso-27001%2Fcase-studies%2FBSI-ISO-IEC-27001-case-study-Thames-Security-UK-EN.pdf%3Fepslanguage%3Den-MY&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSIHowFred12-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSIHowFred12_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">BSI (October 2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Fredrickson-International-EN-UK.pdf?epslanguage=en-MY\" target=\"_blank\">\"How Fredrickson has reduced third party scrutiny and protected its reputation with ISO 27001 certification\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Fredrickson-International-EN-UK.pdf?epslanguage=en-MY\" target=\"_blank\">https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Fredrickson-International-EN-UK.pdf?epslanguage=en-MY<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+Fredrickson+has+reduced+third+party+scrutiny+and+protected+its+reputation+with+ISO+27001+certification&rft.atitle=&rft.aulast=BSI&rft.au=BSI&rft.date=October+2012&rft_id=https%3A%2F%2Fwww.bsigroup.com%2FDocuments%2Fiso-27001%2Fcase-studies%2FBSI-ISO-IEC-27001-case-study-Fredrickson-International-EN-UK.pdf%3Fepslanguage%3Den-MY&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSImplem13-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSImplem13_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">BSI (April 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Legal-Ombudsman-UK-EN.pdf\" target=\"_blank\">\"Implementing best practice and improving client confidence with ISO\/IEC 27001\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Legal-Ombudsman-UK-EN.pdf\" target=\"_blank\">https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Legal-Ombudsman-UK-EN.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Implementing+best+practice+and+improving+client+confidence+with+ISO%2FIEC+27001&rft.atitle=&rft.aulast=BSI&rft.au=BSI&rft.date=April+2013&rft_id=https%3A%2F%2Fwww.bsigroup.com%2FDocuments%2Fiso-27001%2Fcase-studies%2FBSI-ISO-IEC-27001-case-study-Legal-Ombudsman-UK-EN.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSISupport13-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSISupport13_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">BSI (June 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-SVM-UK-EN.pdf\" target=\"_blank\">\"Supporting business growth with ISO\/IEC 27001\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-SVM-UK-EN.pdf\" target=\"_blank\">https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-SVM-UK-EN.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Supporting+business+growth+with+ISO%2FIEC+27001&rft.atitle=&rft.aulast=BSI&rft.au=BSI&rft.date=June+2013&rft_id=https%3A%2F%2Fwww.bsigroup.com%2FDocuments%2Fiso-27001%2Fcase-studies%2FBSI-ISO-IEC-27001-case-study-SVM-UK-EN.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSIInfoview13-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSIInfoview13_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">BSI (2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20160424133846\/https:\/\/www.bsigroup.com\/LocalFiles\/EN-AU\/_Case%20Studies\/BSI%20Infoview%20Case%20Study.pdf\" target=\"_blank\">\"InfoView Case Study\"<\/a> (PDF). Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bsigroup.com\/LocalFiles\/EN-AU\/_Case\" target=\"_blank\">Studies\/BSI Infoview Case Study.pdf the original<\/a> on 24 April 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20160424133846\/https:\/\/www.bsigroup.com\/LocalFiles\/EN-AU\/_Case%20Studies\/BSI%20Infoview%20Case%20Study.pdf\" target=\"_blank\">https:\/\/web.archive.org\/web\/20160424133846\/https:\/\/www.bsigroup.com\/LocalFiles\/EN-AU\/_Case%20Studies\/BSI%20Infoview%20Case%20Study.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=InfoView+Case+Study&rft.atitle=&rft.aulast=BSI&rft.au=BSI&rft.date=2013&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20160424133846%2Fhttps%3A%2F%2Fwww.bsigroup.com%2FLocalFiles%2FEN-AU%2F_Case%2520Studies%2FBSI%2520Infoview%2520Case%2520Study.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSIUsingISO13-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSIUsingISO13_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">BSI (May 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Capgemini-UK-EN.pdf\" target=\"_blank\">\"Using ISO\/IEC 27001 certification to increase resilience, reassure clients and gain a competitive edge\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Capgemini-UK-EN.pdf\" target=\"_blank\">https:\/\/www.bsigroup.com\/Documents\/iso-27001\/case-studies\/BSI-ISO-IEC-27001-case-study-Capgemini-UK-EN.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Using+ISO%2FIEC+27001+certification+to+increase+resilience%2C+reassure+clients+and+gain+a+competitive+edge&rft.atitle=&rft.aulast=BSI&rft.au=BSI&rft.date=May+2013&rft_id=https%3A%2F%2Fwww.bsigroup.com%2FDocuments%2Fiso-27001%2Fcase-studies%2FBSI-ISO-IEC-27001-case-study-Capgemini-UK-EN.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSIIntegrating13-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSIIntegrating13_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">BSI (June 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-22301\/case-studies\/Costain-case-study-UK-EN.pdf\" target=\"_blank\">\"Integrating management systems to improve business performance and achieve sustained competitive advantage\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bsigroup.com\/Documents\/iso-22301\/case-studies\/Costain-case-study-UK-EN.pdf\" target=\"_blank\">https:\/\/www.bsigroup.com\/Documents\/iso-22301\/case-studies\/Costain-case-study-UK-EN.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Integrating+management+systems+to+improve+business+performance+and+achieve+sustained+competitive+advantage&rft.atitle=&rft.aulast=BSI&rft.au=BSI&rft.date=June+2013&rft_id=https%3A%2F%2Fwww.bsigroup.com%2FDocuments%2Fiso-22301%2Fcase-studies%2FCostain-case-study-UK-EN.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-COSOGuidance-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-COSOGuidance_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Committee of Sponsoring Organizations of the Treadway Commission. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.coso.org\/Pages\/erm.aspx\" target=\"_blank\">\"Guidance on Enterprise Risk Management\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.coso.org\/Pages\/erm.aspx\" target=\"_blank\">https:\/\/www.coso.org\/Pages\/erm.aspx<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 06 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Guidance+on+Enterprise+Risk+Management&rft.atitle=&rft.aulast=Committee+of+Sponsoring+Organizations+of+the+Treadway+Commission&rft.au=Committee+of+Sponsoring+Organizations+of+the+Treadway+Commission&rft_id=https%3A%2F%2Fwww.coso.org%2FPages%2Ferm.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CISControls-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CISControls_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Center for Internet Security. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cisecurity.org\/controls\/\" target=\"_blank\">\"CIS Controls\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cisecurity.org\/controls\/\" target=\"_blank\">https:\/\/www.cisecurity.org\/controls\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 06 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CIS+Controls&rft.atitle=&rft.aulast=Center+for+Internet+Security&rft.au=Center+for+Internet+Security&rft_id=https%3A%2F%2Fwww.cisecurity.org%2Fcontrols%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISFStandard-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISFStandard_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Information Security Forum. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.securityforum.org\/tool\/the-isf-standard-good-practice-information-security-2018\/\" target=\"_blank\">\"The ISF Standard of Good Practice for Information Security 2018\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.securityforum.org\/tool\/the-isf-standard-good-practice-information-security-2018\/\" target=\"_blank\">https:\/\/www.securityforum.org\/tool\/the-isf-standard-good-practice-information-security-2018\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+ISF+Standard+of+Good+Practice+for+Information+Security+2018&rft.atitle=&rft.aulast=Information+Security+Forum&rft.au=Information+Security+Forum&rft_id=https%3A%2F%2Fwww.securityforum.org%2Ftool%2Fthe-isf-standard-good-practice-information-security-2018%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrooksonOverview17-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrooksonOverview17_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Brookson, C. (September 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.enisa.europa.eu\/events\/enisa-cscg-2017\/presentations\/brookson\" target=\"_blank\">\"Cybersecurity: Overview of Cybersecurity\"<\/a>. ETSI<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.enisa.europa.eu\/events\/enisa-cscg-2017\/presentations\/brookson\" target=\"_blank\">https:\/\/www.enisa.europa.eu\/events\/enisa-cscg-2017\/presentations\/brookson<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cybersecurity%3A+Overview+of+Cybersecurity&rft.atitle=&rft.aulast=Brookson%2C+C.&rft.au=Brookson%2C+C.&rft.date=September+2017&rft.pub=ETSI&rft_id=https%3A%2F%2Fwww.enisa.europa.eu%2Fevents%2Fenisa-cscg-2017%2Fpresentations%2Fbrookson&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SABSA_SENC-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SABSA_SENC_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">SABSA. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/sabsa.org\/sabsa-nist-framework-project\/\" target=\"_blank\">\"SENC: SABSA Enhanced NIST Cybersecurity Framework\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/sabsa.org\/sabsa-nist-framework-project\/\" target=\"_blank\">https:\/\/sabsa.org\/sabsa-nist-framework-project\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SENC%3A+SABSA+Enhanced+NIST+Cybersecurity+Framework&rft.atitle=&rft.aulast=SABSA&rft.au=SABSA&rft_id=https%3A%2F%2Fsabsa.org%2Fsabsa-nist-framework-project%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IASMEAboutCyber-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IASMEAboutCyber_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">IASME. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/iasme.co.uk\/cyberessentials\/about-cyber-essentials\/\" target=\"_blank\">\"About Cyber Essentials\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/iasme.co.uk\/cyberessentials\/about-cyber-essentials\/\" target=\"_blank\">https:\/\/iasme.co.uk\/cyberessentials\/about-cyber-essentials\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=About+Cyber+Essentials&rft.atitle=&rft.aulast=IASME&rft.au=IASME&rft_id=https%3A%2F%2Fiasme.co.uk%2Fcyberessentials%2Fabout-cyber-essentials%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FraserSite97-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FraserSite97_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Fraser, B., ed. (September 1997). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ietf.org\/rfc\/rfc2196.txt\" target=\"_blank\">\"RFC 2196 - Site Security Handbook\"<\/a>. IETF<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ietf.org\/rfc\/rfc2196.txt\" target=\"_blank\">https:\/\/www.ietf.org\/rfc\/rfc2196.txt<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=RFC+2196+-+Site+Security+Handbook&rft.atitle=&rft.date=September+1997&rft.pub=IETF&rft_id=https%3A%2F%2Fwww.ietf.org%2Frfc%2Frfc2196.txt&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HITRUSTIntro17-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HITRUSTIntro17_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">HITRUST (September 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/hitrustalliance.net\/documents\/csf_rmf_related\/v9\/CSFv9Introduction.pdf\" target=\"_blank\">\"Introduction to the HITRUST CSF, Version 9.0\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/hitrustalliance.net\/documents\/csf_rmf_related\/v9\/CSFv9Introduction.pdf\" target=\"_blank\">https:\/\/hitrustalliance.net\/documents\/csf_rmf_related\/v9\/CSFv9Introduction.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Introduction+to+the+HITRUST+CSF%2C+Version+9.0&rft.atitle=&rft.aulast=HITRUST&rft.au=HITRUST&rft.date=September+2017&rft_id=https%3A%2F%2Fhitrustalliance.net%2Fdocuments%2Fcsf_rmf_related%2Fv9%2FCSFv9Introduction.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NERCSumm14-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NERCSumm14_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Vinson & Elkins (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.velaw.com\/uploadedfiles\/vesite\/resources\/summarycipversion5standards2014.pdf\" target=\"_blank\">\"Summary of CIP Version 5 Standards\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.velaw.com\/uploadedfiles\/vesite\/resources\/summarycipversion5standards2014.pdf\" target=\"_blank\">https:\/\/www.velaw.com\/uploadedfiles\/vesite\/resources\/summarycipversion5standards2014.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 February 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Summary+of+CIP+Version+5+Standards&rft.atitle=&rft.aulast=Vinson+%26+Elkins&rft.au=Vinson+%26+Elkins&rft.date=2014&rft_id=https%3A%2F%2Fwww.velaw.com%2Fuploadedfiles%2Fvesite%2Fresources%2Fsummarycipversion5standards2014.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSALandscape-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OSALandscape_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">OSA. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.opensecurityarchitecture.org\/cms\/foundations\/osa-landscape\" target=\"_blank\">\"OSA Landscape\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.opensecurityarchitecture.org\/cms\/foundations\/osa-landscape\" target=\"_blank\">http:\/\/www.opensecurityarchitecture.org\/cms\/foundations\/osa-landscape<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OSA+Landscape&rft.atitle=&rft.aulast=OSA&rft.au=OSA&rft_id=http%3A%2F%2Fwww.opensecurityarchitecture.org%2Fcms%2Ffoundations%2Fosa-landscape&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CYSEC_GPG13Exec-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CYSEC_GPG13Exec_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">CYSEC. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/gpg13.com\/executive-summary\/\" target=\"_blank\">\"GPG13 Executive Summary\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/gpg13.com\/executive-summary\/\" target=\"_blank\">http:\/\/gpg13.com\/executive-summary\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 March 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GPG13+Executive+Summary&rft.atitle=&rft.aulast=CYSEC&rft.au=CYSEC&rft_id=http%3A%2F%2Fgpg13.com%2Fexecutive-summary%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance. Some original references had broken URLs; this version updates them to functional URLs. In one case, an archived version of the article had to be used.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20200707204339\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.490 seconds\nReal time usage: 0.918 seconds\nPreprocessor visited node count: 21730\/1000000\nPreprocessor generated node count: 34863\/1000000\nPost\u2010expand include size: 144136\/2097152 bytes\nTemplate argument size: 57129\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 434.952 1 - -total\n 83.42% 362.853 1 - Template:Reflist\n 69.91% 304.095 32 - Template:Citation\/core\n 64.51% 280.605 29 - Template:Cite_web\n 8.95% 38.916 3 - Template:Cite_journal\n 8.43% 36.652 1 - Template:Infobox_journal_article\n 8.06% 35.069 1 - Template:Infobox\n 5.47% 23.776 80 - Template:Infobox\/row\n 3.59% 15.599 36 - Template:Citation\/make_link\n 1.28% 5.570 3 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11400-0!*!0!!en!5!*!math=5 and timestamp 20200707204339 and revision id 37680\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study\">https:\/\/www.limswiki.org\/index.php\/Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","5ed990f0be9ad0fbc7220eb00577224e_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/13\/Fig1_Ibrahim_JoSupercomp2018_74-10.png"],"5ed990f0be9ad0fbc7220eb00577224e_timestamp":1594154618,"2f4c8b4d90b1c8731d79d7ff410d7c37":{"type":"chapter","title":"1. Cybersecurity","key":"2f4c8b4d90b1c8731d79d7ff410d7c37"}},"link":"https:\/\/www.limswiki.org\/index.php\/Book:LIMSjournal_-_Spring_2020","price_currency":"","price_amount":"","book_size":"","download_url":"https:\/\/www.limsforum.com?ebb_action=book_download&book_id=91415","language":"","cta_button_content":"","toc":[{"type":"chapter","name":"1. Cybersecurity","id":"2f4c8b4d90b1c8731d79d7ff410d7c37","children":[{"type":"article","name":"A security review of local government using NIST CSF: A case study (Ibrahim et al. 2018)","id":"5ed990f0be9ad0fbc7220eb00577224e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:A_security_review_of_local_government_using_NIST_CSF:_A_case_study"},{"type":"article","name":"Information technology and medical technology personnel\u2019s perception regarding segmentation of medical devices: A focus group study (Johansson et al. 2020)","id":"6fa400ac6b9e1784eed63829a3731260","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Information_technology_and_medical_technology_personnel%E2%80%99s_perception_regarding_segmentation_of_medical_devices:_A_focus_group_study"}]},{"type":"chapter","name":"2. Laboratory informatics and bioinformatics","id":"669790d43b8cd47cddb31687a18f840b","children":[{"type":"article","name":"One tool to find them all: A case of data integration and querying in a distributed LIMS platform (Grand et al. 2019)","id":"45b5e684bad1c44cd9a1cb43bb6b311e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:One_tool_to_find_them_all:_A_case_of_data_integration_and_querying_in_a_distributed_LIMS_platform"},{"type":"article","name":"ROBOT: A Tool for Automating Ontology Workflows (Jackson et al. 2019)","id":"2351ba0d1aa807936bc9922c2403b7f2","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:ROBOT:_A_Tool_for_Automating_Ontology_Workflows"},{"type":"article","name":"From command-line bioinformatics to bioGUI (Joppich and Zimmer 2019)","id":"41e81da03f2f60115a39008f3e196345","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:From_command-line_bioinformatics_to_bioGUI"},{"type":"article","name":"Design and implementation of a clinical laboratory information system in a low-resource setting (Mtonga et al. 2019)","id":"eb76e424fc635068c27be185a754eac7","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Design_and_implementation_of_a_clinical_laboratory_information_system_in_a_low-resource_setting"},{"type":"article","name":"Universal LIMS-based platform for the automated processing of cell-based assays (Schmieder et al. 2019)","id":"c28779d8f7eda776acc43ead069c1ea8","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Universal_LIMS-based_platform_for_the_automated_processing_of_cell-based_assays"}]},{"type":"chapter","name":"3. Laboratory medicine and analysis","id":"a2e263afc6b7d71814ba54b573155973","children":[{"type":"article","name":"Project management in laboratory medicine (Lippi and Mattiuzi 2019)","id":"9d43e21cd08ee50665979d490d94d5ef","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Project_management_in_laboratory_medicine"},{"type":"article","name":"Ultra-high-performance liquid chromatography coupled with quadrupole-Orbitrap high-resolution mass spectrometry for multi-residue analysis of mycotoxins and pesticides in botanical nutraceuticals (Narv\u00e1ez et al. 2020)","id":"3eda7d7fb752e96c7816c5992924922e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Ultra-high-performance_liquid_chromatography_coupled_with_quadrupole-Orbitrap_high-resolution_mass_spectrometry_for_multi-residue_analysis_of_mycotoxins_and_pesticides_in_botanical_nutraceuticals"},{"type":"article","name":"Point-of-care RNA-based diagnostic device for COVID-19 (Yang et al. 2020)","id":"c3c7e4673674f7b161f8fb6199af909a","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Point-of-care_RNA-based_diagnostic_device_for_COVID-19"}]},{"type":"chapter","name":"4. Open source paradigm","id":"21339472a2568d169a14cebb4436e2bb","children":[{"type":"article","name":"What is the \"source\" of open-source hardware? (Bonvoisin et al. 2017)","id":"3ab9770e7fca9810f5573456215e3853","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_%22source%22_of_open-source_hardware%3F"},{"type":"article","name":"Leveraging conservation action with open\u2010source hardware (Hill et al. 2019)","id":"d14ab23a070a6dedd70edd1f31559d55","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Leveraging_conservation_action_with_open%E2%80%90source_hardware"}]}],"settings":{"show_cover":1,"show_title":1,"show_subtitle":0,"show_full_title":1,"show_editor":1,"show_editor_pic":1,"show_publisher":1,"show_language":1,"show_size":1,"show_toc":1,"show_content_beneath_cover":1,"toc_links":"disabled","cta_button":"1","content_location":"1","log_in_msg":"<span><\/span> Please log in to read online.","cover_size":"medium"},"title_image":"https:\/\/s3.limsforum.com\/www.limsforum.com\/wp-content\/uploads\/Fig1_Mtonga_AfricanJofLabMed2019_8-1.jpg"}}
LIMSjournal - Spring 2020
Volume 6, Issue 1
Editor: Shawn Douglas
Publisher: LabLynx Press
Copyright LabLynx Inc. All rights reserved.