{"ID":78043,"post_author":"2","post_date":"2018-12-14 13:14:46","post_date_gmt":"0000-00-00 00:00:00","post_content":"","post_title":"LIMSjournal - Summer 2017","post_excerpt":"","post_status":"draft","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"","to_ping":"","pinged":"","post_modified":"2018-12-14 13:14:46","post_modified_gmt":"2018-12-14 18:14:46","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.limsforum.com\/?post_type=ebook&p=78043","menu_order":0,"post_type":"ebook","post_mime_type":"","comment_count":"0","filter":"","_ebook_metadata":{"enabled":"on","private":"0","guid":"4D98F986-C5BB-422D-B305-9457835D59AB","title":"LIMSjournal - Summer 2017","subtitle":"Volume 3, Issue 2","cover_theme":"nico_21","cover_image":"https:\/\/www.limsforum.com\/wp-content\/plugins\/rdp-ebook-builder\/pl\/cover.php?cover_style=nico_21&subtitle=Volume+3%2C+Issue+2&editor=Shawn+Douglas&title=LIMSjournal+-+Summer+2017&title_image=https%3A%2F%2Fs3.limsforum.com%2Fwww.limsforum.com%2Fwp-content%2Fuploads%2FFig3_DAnca_NatHazEarth2017_17-2.png&publisher=LabLynx+Press","editor":"Shawn Douglas","publisher":"LabLynx Press","author_id":"26","image_url":"","items":{"bce85c098ea6958c92b6dcce94e42565_type":"article","bce85c098ea6958c92b6dcce94e42565_title":"Neuroimaging, genetics, and clinical data sharing in Python using the CubicWeb framework (Grigis et al. 2017)","bce85c098ea6958c92b6dcce94e42565_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework","bce85c098ea6958c92b6dcce94e42565_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Neuroimaging, genetics, and clinical data sharing in Python using the CubicWeb framework\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nNeuroimaging, genetics, and clinical data sharing in Python using the CubicWeb frameworkJournal\n \nFrontiers in NeuroinformaticsAuthor(s)\n \nGrigis, Antoine; Goyard, David; Cherbonnier, Robin; Gareau, Thomas; Papadopoulos Orfanos, Dimitri;\r\nChauvat, Nicolas; Di Mascio, Adrien; Schumann, Gunter; Spooren, Will; Murphy, Declan; Frouin, VincentAuthor affiliation(s)\n \nUniversit\u00e9 Paris-Saclay, Logilab, King\u2019s College London, F. Hoffmann-La Roche PharmaceuticalsPrimary contact\n \nEmail: antoine dot grigis at cea dot frEditors\n \nMarcus, DanielYear published\n \n2017Volume and issue\n \n11Page(s)\n \n18DOI\n \n10.3389\/fninf.2017.00018ISSN\n \n1662-5196Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/journal.frontiersin.org\/article\/10.3389\/fninf.2017.00018\/fullDownload\n \nhttp:\/\/journal.frontiersin.org\/article\/10.3389\/fninf.2017.00018\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Materials and methods \n\n3.1 CubicWeb overview \n3.2 Structured data upload service \n3.3 Collaborative quality control service \n3.4 Publication service \n\n3.4.1 A dedicated structure for imaging genomics questionnaire data \n3.4.2 Efficient data selection and download tool: The data shopping cart mechanism \n3.4.3 The transfer of the shopping cart content: Data download \n3.4.4 Access rights mechanism \n3.4.5 The unified insertion procedure \n\n\n3.5 A transverse Python module to remotely connect a CubicWeb DSS \n\n\n4 Results \n5 Discussion and conclusion \n\n5.1 Lightweight solution for data sharing \n5.2 A PIx Swiss knife \n5.3 Future directions \n\n\n6 Author contributions \n7 Conflict of interest statement \n8 Acknowledgments \n9 Supplementary material \n10 References \n11 Notes \n\n\n\nAbstract \nIn neurosciences or psychiatry, the emergence of large multi-center population imaging studies raises numerous technological challenges. From distributed data collection, across different institutions and countries, to final data publication service, one must handle the massive, heterogeneous, and complex data from genetics, imaging, demographics, or clinical scores. These data must be both efficiently obtained and downloadable. We present a Python solution, based on the CubicWeb open-source semantic framework, aimed at building population imaging study repositories. In addition, we focus on the tools developed around this framework to overcome the challenges associated with data sharing and collaborative requirements. We describe a set of three highly adaptive web services that transform the CubicWeb framework into a (1) multi-center upload platform, (2) collaborative quality assessment platform, and (3) publication platform endowed with massive-download capabilities. Two major European projects, IMAGEN and EU-AIMS, are currently supported by the described framework. We also present a Python package that enables end users to remotely query neuroimaging, genetics, and clinical data from scripts.\nKeywords: web service, data sharing, database, neuroimaging, genetics, medical informatics, Python\n\nIntroduction \nHealth research strategies using neuroimaging have shifted in recent years: the focus has moved from patient care only, to a combination of patient care and prevention. In the case of neurodegenerative and psychiatric diseases, this drives the creation of increasingly numerous massive imaging studies, also known as population imaging (PI) surveys.[1][2] It should be noticed that PI studies no longer consist of image data only. The recent wide availability of high-throughput genomics has augmented the subject data with genetics, epigenetics, and functional genomics. Likewise, the standardization of personality, demographics, and deficit tests in psychiatry facilitates the acquisition of clinical\/behavioral records to enrich the subject data in large population studies. Moreover, PI studies now classically encompass more than one single imaging session per subject and cover multiple-time point heterogeneous experiments. Ultimately, these studies with complex imaging and extended data (PIx) require multi-center acquisitions to build a large target population.\nA regular PIx infrastructure has to cover the following three main topics: (1) data collection, (2) quality control (QC) with data processing, and (3) data indexing and publication with controlled data sharing mechanisms. Furthermore, PIx infrastructures must evolve during the life cycle of a population imaging project, and they must also be resilient to extreme evolutions of the data content and management. In the projects we manage, we experience several extreme evolutions. The first kind of evolution may affect the published dataset such as adding a new modality for all subjects, a new time point or a new subcohort. Second, the amount of data requested evolves dramatically as the project consortium gets enlarged.[3] Finally, internal ontologies have to evolve constantly in order to match the ongoing initiatives on interoperability.[4][5]\nSeveral existing open-source frameworks support one or several of the described topics, sometimes only for one specific data type. We propose in the following a brief overview of existing systems. Some of these systems have also been reviewed by Nichols and Pohl.[6] IDA[7] is a neuroimaging data repository and management system that supports data collection (topic one) and data sharing (topic three). With this system, the published datasets can be searched using automatically extracted metadata. The XNAT framework[8] is widely used for neuroimaging data and supports all the PIx infrastructure topics, focusing on tools to pipeline, and to audit the processing of image data (topic two). The LORIS[9] and NiDB[10] frameworks represent a significant effort to account for multimodal data involved in PIx studies. These frameworks, although addressing all the required topics, mainly support neuroimaging data. Openclinica[11] and REDCap[12] facilitate the collection of electronic data such as eCRF or questionnaires and are recognized in projects of various sizes that support data collection (topic one). Likewise, laboratory information management systems were developed for the collection of genomic measurements such as SIMBioMS.[13] Finally, the COINS framework brings essential tools for multimodal data support and, more interestingly, emphasizes the importance of providing sharing tools (topics one and three).[14]\nThe two European studies we manage require a tailored PIx infrastructure. Existing frameworks neither completely handle the diversity of our PIx requirements and project life cycle nor provide efficient tools to collect, check the quality of, and publish evolving data. Additional developments were required for building such complete infrastructure. We based these developments on a more general framework than the dedicated applications described above. In collaboration with Logilab company (Logilab SA, Paris, France), we developed three highly adaptive web services, based on the CubicWeb (CW) pure-Python framework, aimed at creating a (1) multi-center upload platform, (2) collaborative quality assessment platform, and (3) publication platform with massive-download features.[15] These developments were originally instituted for IMAGEN and EU-AIMS projects in order to host their data about mental health in adolescents[16] and autism[17], respectively. The corresponding studies require key features such as upload\/browse published data from the web, dynamic selection and filtering of displayed data, support for flexible download operations, high-level request language, multilevel access rights, remote data access, remote user access rights management, collaborative QC, and interoperability.\n\nMaterials and methods \nThe three services described in the introduction were handled in distinct developments. The next sub-section presents the CW framework capabilities, followed by introductions in the second and fourth sub-sections to the upload and publication web services through which the tailored requirements of PIx studies are satisfied. Furthermore, we describe in the third sub-section a collaborative rating web service that helps users to assess the data quality, and in the final sub-section a Python API that remotely queries these web services.\n\nCubicWeb overview \nAll the implemented services are based on the CW framework.[15] We choose a high level pure-Python framework that bridges web technologies and database engines. This choice was also based on the expertise and experience of people from our laboratory and a tight collaboration with Logilab.[18][19] CW distribution is organized in a core part and a set of basic Python modules, referred to as cubes, which can be used to efficiently generate web applications. The core of the CW framework, developed under the LGPL license, is constructed from well-established technologies (SQL, Python, web technologies such as HTML5 and Javascript). The main characteristics of the CW framework are given as follows:\n\n1. CW defines its data model with Python classes and automatically generates the underlying database structure.\n2. The queries are expressed with the RQL language, which is similar to W3C\u2019s SPARQL.[20] All the persistent data are retrieved and modified using this language.\n3. CW implements a mechanism that exposes information in several ways, referred to as views. This mechanism implements the classical model-view-controller software architecture pattern. Defined in Python, the views are applied to query results and can produce HTML pages and\/or trigger external processes. The separation of queries and views offers major advantages: first, the same data selection may have several web representations, and second, retrieved data can be exported in several other formats without modifying the underlying data storage.\n4. All the views and triggers are recorded in a registry and are automatically selected depending on the current context, which is inferred from the type of data returned by the RQL.\n5. Thanks to the semantic nature of CW, all developments inherit the possibility to follow existing or emerging ontologies, thereby facilitating sharing, access, and processing.\n6. CW has a security system that grants fine-grained access to the data. This system is similar to the row-level security and policies available in the most recent versions of PostgreSQL, and links access rights to entities\/relations in the schema. Each entity type has a set of attributes and relations, and permissions that define who can add, read, update, or delete such an entity and associated relations.\n7. CW may run either as a standalone application or behind an Apache front server. We refer to both settings as a data sharing service (DSS) (cf. Figure 1).\n8. CW can be configured to run with various database engines. For the best performance, PostgreSQL is recommended.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Architecture of a CubicWeb data sharing service (DSS) integrated in an Apache platform with LDAP. The business logic cubes provide a schema that can be instantiated in the database management system (DBMS: red puzzle piece). The system cubes ensure low-level system interactions (green puzzle piece), and the application cube proposes a web user interface (blue puzzle piece). End users access the database content through a web browser, a Python API scripting the DSS or an FTP solution, where virtual folders (acting as filters on the central repository) are proposed for download.\n\n\n\nStarting from the basic CW distributions, our suite of services is composed of an assembly of Python modules, also referred to as cubes. The Python language is widely used in scientific communities and facilitates interfacing with major or emerging processing tools such as Nipype[21], Biopython[22], Nilearn[23], and Morphologist.[24] Application cubes, built over system cubes, and business logic cubes can be distinguished. The system cubes ensure interactions with the operating system and middleware. For example, they connect to LDAP for user credentials and information or invoke FUSE[25] as a module to construct virtual file systems in a user repository for downloading. The business logic cubes essentially provide the database schema and the application cubes define the access rights and the web interface.\nAmong the available Python-based frameworks, we chose CW. A major advantage of CW is the RQL language which brings end users a query interface adapted for PIx data sharing. It simplifies and improves the user experience in searching for custom datasets. RQL also avoids the use of a complicated object relational mapper (ORM), is focused on browsing relations, and allows requesting several DSS at once. The semantic nature of this request language requires the user to know only about the used data model defined as a graph (nothing about the underlying low-level relational model). This data model simplification and the expressiveness of RQL help users writing custom requests, while most of existing DSS do not expose a query language but offer a limited predefined number of operations that can be carefully designed to be efficient (e.g., RESTful APIs). Criticisms against systems exposing a query request language to the end users emphasize a risk of denial of service. To avoid this issue (i.e., overloading the server with arbitrary complex requests), CW allows limitation of usable resources (RAM per request, CPU per request, number of requests per user, CPU time per request). We believe that users should be able to select and download only what they specifically need using a query request language. This avoids filtering the data locally and saves the bandwidth.\n\nStructured data upload service \nIn PIx studies, massive and complex data are gathered from multiple data acquisition centers or devices (topic one). Each collected dataset must be mapped with definitions that follow consensus representation rules. Those definitions are grouped in data dictionaries that ideally follow standards[26], but they are mainly manufacturer- and\/or site-specific. Thus, an efficient and versatile tool is required for mapping the different data dictionaries during the collection process.\nLeveraging those ideas, we implemented a flexible upload mechanism, a system cube named rql_upload[27] and provided a web frontend by integrating this cube with the application cube named PIWS[28] (Population Imaging Web Service, cf. Figure 1). Based on a CW feature that allows database completion through online HTML forms, these two cubes were developed to collect, in a DSS, both raw data and metadata. CW also enables the customization of triggers that determine the integrity of the uploaded data: synchronous and asynchronous validation filters can be specified and applied to each upload dataset. The upload proceeds as follows (cf. Figure 2):\n\n1. Synchronous validations are applied to each form field (e.g., to check the extension of a file or the structure of an Excel table). If the validation filtering fails, then the web form is refreshed and an adapted feedback is displayed.\n2. After synchronous validation, all the uploaded raw data\/metadata are stored in generic entities and a \"Quarantine\" status is set. To avoid cluttering of the database and to ease file manipulation, files are stored in the central repository but remain accessible through the database. File hashes are automatically computed and indexed in order to assess data integrity.\n3. To update the upload status from \"Quarantine\" to \"Rejected\"\/\"Validated,\" automatic asynchronous validations can be configured in the service as looping tasks. Those validation filters are project and\/or data and\/or upload specific and generate adapted feedback for users and data managers.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Illustration of the upload process. The (A) syntax of a form description JSON file, (B) corresponding web form as presented to users (here an error message returned by synchronous validation is displayed in the top red box), (C) \u201cQuarantine\u201d status, and (D) \u201cValidated\u201d status (obtained after asynchronous validation) as displayed to users: note that no feedback is shown here.\n\n\n\nMoreover, any entity or relation may be endowed with access permission rules.[15] Based on the CW security mechanisms, a customized security model was implemented for our upload DSS (it can be extended later). Only specific groups have the authorization to upload, and users can only access the uploads, which they are interested in. The customization of these core features allowed the creation of an upload web service that is completely described in a single JSON file. This file links the web form fields with customized or CW-internal controllers that manage the type of data to be collected.\n\nCollaborative quality control service \nOwing to the large amount of data gathered\/analyzed in PIx studies, we must consider more sophisticated operating procedures than simple quality controls (QCs), where datasets are usually only rated once by a handful of individuals. This issue can be addressed by implementing a web-based collaborative quality control process that will also remove the bias introduced by isolated raters (topic two). Moreover, for the studies we manage, we also added controlled vocabulary description to the ratings.\nWe achieve these goals by implementing a flexible collaborative rating mechanism, i.e., an application cube named zeijemol.[29] As in the previous sub-section, a collaborative quality control DSS is entirely described in a single JSON file. This file consists, on the one hand, of the list of elements that will be rated (e.g., a Nifti image, a FreeSurfer segmentation, or a motion curve in a diffusion sequence of an individual) and, on the other hand, related quality indicators (e.g., binary good\/bad, controlled vocabulary, scaled rating). Each element is displayed by one of the embedded viewers such as triplanar view or mesh rendering (cf. Figure 3). The QC results are stored directly in the database.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. The collaborative quality control web service of a FreeSurfer segmentation element of one subject. (A) the quality indicators (in this case, a controlled vocabulary with an accept\/prescribe manual edit\/reject decision and an optional check-box justification), (B) a triplanar view of the white and pial surfaces overlayed on the anatomical image, and (C) the white and pial meshes with statistical indicators.\n\n\n\nThe emergence of such DSS will allow machine learning techniques to learn new classifiers to automatized the quality control task. The QC scores may also be directly used as prior knowledge during the analysis stage.\n\nPublication service \nIn PIx studies, data collection and QC are followed by data anonymization, ordering, and analysis. Ultimately, data are made available to the acquisition partners or the scientific community (topic three). While browsing the database content through the web interface, users expect to be able to download the displayed files as well as the data description and rich links between the data, also referred to as metadata. An intuitive and reliable sharing mechanism is therefore crucial as large amounts of heterogeneous evolving data must be provided. Furthermore, for the studies we manage, access rights are split along time points, scan types, questionnaires, or questions to match the consortia multilevel access permissions.\nTherefore, we implemented a system cube named rql_download[30] and provided a web frontend by integrating this cube with PIWS[28] whenever it was used in a publication service (cf. Figure 1). The rql_download cube converts the result of any RQL query into files on a virtual file system that, in turn, can be accessed through a secured file transfer protocol (sFTP) (cf. Figure 1). The following five sub-sections introduce: (1) the business logic cubes used to describe the neuroimaging genetics data and metadata and the relationship between these data; (2) how users can save the content of their current search from the DSS web interface; (3) two approaches of rql_download, based on two basic softwares (FUSE or Twisted), that give users access to their saved searches, as well as the pros and cons of both; (4) a suitable strategy for setting user rights from the CW security system; and (5) a descriptive data insertion mechanism, as a set of Python scripts.\n\nA dedicated structure for imaging genomics questionnaire data \nThe database schema was developed for handling multi-time point\/multimodal datasets in the brainomics business logic cube.[31] This schema supports general information such as subject data and associated metadata (age, handedness, sex, etc.), acquisition center definitions, multimodal imaging datasets, clinical\/behavioral records, processed data, and some genomic concepts (including chromosomes, genes, SNPs, or genomic platforms). An excerpt of the produced schema is shown in Figure 4.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. A snippet of the schema used in a publication DSS. We see from the green boxes that all entities are related to an \u201cAssessment\u201d entity through an \u201cin_assessment\u201d relation. This behavior is inherited from the access rights described in the fourth sub-section.\n\n\n\nEfficient data selection and download tool: The data shopping cart mechanism \nWhen an RQL query result set is returned by the DSS, the most adapted view is automatically selected, and facets are attached to each web page, thereby providing filtering rules. Facets allow interactive and graphical search refinements in accordance with selected attributes (e.g., sex or handedness filter for a subject result set). The developed shopping cart mechanism serves to save the user searches that consist of data, possibly large files, and metadata. This mechanism and the facet filtering are smoothly integrated: activating a filter option from the web interface automatically updates the search query result set, and thus, the list of files that will be dropped for download (cf. Figure 5). The data added to one cart has an expiration date that can be configured in the service. Convenient access rights are set: users can only access their own searches. For the sake of the EU-AIMS project hosted in our laboratory, a video explaining the data shopping cart mechanism is available (.mp4 file).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Illustration of the download process via the proposed shopping cart mechanism. (A) the facet filter bar when all the scans (\u201cScan\u201d entities) are requested (as highlighted in bold, the user has selected only the \u201cFU2\u201d time point and the diffusion MRI \u201cDTI\u201d scans), (B) the view corresponding to the filtered dataset, (C) add this new search to the cart (by activating these filtering options, the save RQL path search will be automatically updated), (D) a new search has been created, and (E) the download of the search and associated files as presented in FileZilla.\n\n\n\nThe transfer of the shopping cart content: Data download \nWhen saved, the cart content is made available as virtual files and folders. A major advantage of the developed solution is that data compression or duplication is avoided, and that in turn requires no extra load for the publication DSS. Data download operations are delegated to sFTP servers to ensure secure transfers. The sFTP is standard and supported by numerous client software on most systems.\nTwo approaches are implemented in the rql_download cube that can be selected by configuration settings:\n\n1. FUSE virtual folders: For each search, the system builds a list of files to be downloaded and subsequently creates a virtual FUSE directory acting as a filter on the central repository. The user can only see subsets of files\/directories corresponding to his queries built in accordance with his access rights. Finally, the system delegates the data transfers to the sFTP server. The major advantage of this approach is the use of the standard sFTP port. However, additional system level configurations are required during the installation of the DSS in order to set the user home directories and system accounts.\n2. Twisted server: This approach is characterized by a Python process that creates a Twisted[32] event-driven networking server, retrieves all the searches in the database, and exposes the files via sFTP through the created server. Again, this process acts as a filter on the central repository where a user only sees a subset of files\/directories. In this case, the authentication and file transfers are directly operated by CW. The major advantage of this strategy is that no system-level configuration is required. However, listening on a non-default sFTP port, which could lead to firewall issues, is sometimes required.\nAccess rights mechanism \nIn the CW security model, any entity or relation may be endowed with permission rules. To fulfill consortia's criteria, we propose an operational setup of the CW security model for our publication DSS. We built our security model around \"pivotal entities\" rather than specifying rights on all entities. Pivotal entities are those on which access rights are defined, and they are related to all entities that must be covered by the security model through a specific relation (the \"in_assessment\" relation in Figure 4). Each time an entity covered by the security model is requested, the system automatically requests its related pivotal entity and propagates the corresponding access rights.\n\nThe unified insertion procedure \nA unified insertion module is provided as a set of Python scripts to insert neuroimaging, genomic, and clinical data such as scans, genomic measures, questionnaires, and processing steps. These scripts were helpful in efficiently managing the large amount of evolving data in our projects. The indexed data are uniformly organized according to the schema structure and thus take advantage of all the aforementioned developments (e.g., shopping cart mechanism and security model of previous sub-sections, and common renderings cf. Figure 6). Generating such a DSS with these scripts can be performed without specific CW knowledge. Indeed, only a rich description of the data to be published is required as a set of Python dictionary objects.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. Summary views of the database status. Global information, for example the (A) gender or (B) handedness distributions, (C) acquisition status, and (D) age distribution, or longitudinal information, such as (E) the answers of subject2 to specific questions across the study time points.\n\n\n\nA transverse Python module to remotely connect a CubicWeb DSS \nWith the aforementioned capabilities of the DSS, a user manually selects and downloads data through graphical interfaces in order to analyze them locally (as discussed in the previous sub-section). In the case of an evolving DSS, the downloaded data must be regularly updated, and this manual process becomes time consuming and error prone when large and heterogeneous data are considered. Moreover, the metadata, such as quality scores, used to specify the dataset to download are also likely to change. Therefore, to achieve the analysis of up-to-date data stored in a DSS, direct programmatic interaction with the DSS is recommended. In the neuroimaging and neuroscience communities, data are typically analyzed by using Python scripts. Classically, the systems provide RESTful web services such as XNAT, with a Python API.[33] Inheriting from the RQL request language, our publication DSS (cf. previous sub-section) offers a rich interface to access the data.\nWe provide a regular Python module, named cwbrowser[34], that implements a Python API to connect and send RQL to a remote DSS based on the CW framework. This module is completely independent of CW (no CW installation required) and similar to the CW distribution cwclientlib cube. A publication DSS, as described in the previous sub-section, can be requested by the cwbrowser module that embeds the previously described data selection and shopping cart capabilities. It automatically fills and saves a shopping cart from a custom RQL request, downloads the associated virtual directories onto the local file system, and returns the complete requested dataset. The returned dataset contains metadata stored in the DSS such as subject sex or quality scores as well as the path to the downloaded directories. These resources are organized following the DSS layout of files and folders. The users will get the same local tree which will help in writing sharable analysis scripts.\n\nResults \nOur laboratory operates several DSS for the IMAGEN project about mental health in adolescents[16] and the EU-AIMS project about autism.[17] Other DSS are currently under development to support new and ongoing initiatives. Note that the access to both IMAGEN and EU-AIMS datasets is (to date) restricted.\nIn the IMAGEN project, 2,000 subjects are monitored on at least two visits (the third follow-up is underway). T1, T2, FLAIR, DWI, B0, task fMRI, and resting-state fMRI scans are acquired, as well as clinical\/behavioral records, genotyping, gene expression, and methylation. A publication DSS at https:\/\/imagen2.cea.fr\/database enables us to share more than 37,000 scans, 32,000 processing results, and 16 million distinct variables. In the near future, an upload DSS will allow us to collect a new time point.\nIn the EU-AIMS project, 1,500 subjects (from six months to 30 years old) are monitored on several visits through two distinct studies. T1, T2, FLAIR, DWI, B0, task fMRI, resting-state fMRI, and spectroscopy scans are acquired, as well as clinical\/behavioral records, EEG, eye-tracking, gene expression, and methylation. An upload DSS at https:\/\/eu-aims.cea.fr\/database provides the means for collecting this data from 10 centers across Europe. In addition, a collaborative quality check DSS at https:\/\/eu-aims.cea.fr\/qc allows us to assess the uploaded data quality, and a publication DSS at https:\/\/eu-aims.cea.fr\/data_repository enables us to share more than 13,000 scans, 12,000 processing results, and 15 million distinct variables.\n\nDiscussion and conclusion \nLightweight solution for data sharing \nWe developed a novel and lightweight PIx software infrastructure exclusively based on the CW framework. We offer a suite of CW tools that facilitates the creation of a DSS. The system delivers the data to users based on the principle of \"what you see is what you get\": users define their datasets of interest by browsing the database. Thanks to the RQL and the developed Python API, remote query of a DSS is easy and intuitive. In this environment, core features such as the schema definition, the web rendering of the database content, and the semantic request language are provided by a few lines of Python code at the heart of the CW framework. Our DSS can use any database engine, offers an access permission mechanism, and can be smoothly integrated with the standard Apache environment. Moreover the CW framework relies on a large community of developers led by Logilab.\n\nA PIx Swiss knife \nCW is well suited for all the scenarios one can face in a PIx project. For instance, in the projects we manage, we also provided a CW-based service to allow a collaborative moderation of user access to the different DSS. This service enables the consortium review boards to assign the relevant access rights to new or existing users. It is restricted to a few members and enables the user account administration of an upload, collaborative QC, and publication DSS.\n\nFuture directions \nOur developments inherit the web semantic capabilities embedded in the CW framework. Thanks to this key feature, numerous problems of interoperability can be efficiently tackled using emerging ontologies and standards in neuroinformatics, neurosciences, and bioinformatics, such as the NIDM standard for data exchange[35], the Cognitive Atlas Ontology[36], and OntoNeuroLOG[37] for data annotation, or the Bio2RDF for the federation of large datasets using open-source semantic web technologies.[38] The annotation of our datasets, with respect to these ontologies, is ongoing. Ultimately, should all DSS follow standard ontologies, RQL would provide new cross-project querying possibilities. Although the CW framework is already used successfully in several commercial applications, it would be interesting to evaluate the CW framework performances on our DSS with Logilab dedicated tools.\n\nAuthor contributions \nAG developed the cubes, performed its deployment, and maintained the online repositories. DG, DO, TG, and RC tested the proposed application and used it in two European projects (IMAGEN, EU-AIMS). NC and AM developed the CubicWeb framework. VF, GS, WS, and DM initiated and supervized the projects. All authors contributed to the manuscript.\n\nConflict of interest statement \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nAcknowledgments \nThe research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant agreement no. 115300 (EU-AIMS), resources of which are composed of financial contribution from the European Union\u2019s Seventh Framework Programme (FP7\/2007-2013) and EFPIA companies\u2019 in kind contribution.\n\nSupplementary material \nCodes are distributed under the terms of the CeCILL-B license, as published by the CEA-CNRS-INRIA. Refer to the license file or to http:\/\/www.cecill.info\/licences\/Licence_CeCILL-B_V1-en.html for details. Codes are freely accessible on github https:\/\/github.com\/neurospin. The DSS we are in charge of can be reached at https:\/\/imagen2.cea.fr and https:\/\/eu-aims.cea.fr.\n\nReferences \n\n\n\u2191 Hurko, O.; Black, S.E.; Doody, R. et al. (2012). \"The ADNI Publication Policy: Commensurate recognition of critical contributors who are not authors\". NeuroImage 59 (4): 4196\u20134200. doi:10.1016\/j.neuroimage.2011.10.085. PMC PMC3676932. PMID 22100665. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3676932 .   \n\n\u2191 Poldrack, R.A.; Gorgolewski, K.J. (2014). \"Making big data open: Data sharing in neuroimaging\". Nature Neuroscience 17 (11): 1510\u20137. doi:10.1038\/nn.3818. PMID 25349916.   \n\n\u2191 Gorgolewski, K.J.; Varoquaux, G.; Rivera, G. et al. (2015). \"NeuroVault.org: A web-based repository for collecting and sharing unthresholded statistical maps of the human brain\". Frontiers in Neuroinformatics 9: 8. doi:10.3389\/fninf.2015.00008. PMC PMC4392315. PMID 25914639. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4392315 .   \n\n\u2191 Scheufele, E.; Aronzon, D.; Coopersmith, R. et al. (2014). \"tranSMART: An Open Source Knowledge Management and High Content Data Analytics Platform\". AMIA Joint Summits on Translational Science 2014: 96\u2013101. PMC PMC4333702. PMID 25717408. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4333702 .   \n\n\u2191 Gorgolewski, K.J.; Auer, T.; Calhoun, V.D. et al. (2016). \"The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments\". Scientific Data 3: 160044. doi:10.1038\/sdata.2016.44. PMC PMC4978148. PMID 27326542. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4978148 .   \n\n\u2191 Nichols, B.N.; Pohl, K.M. (2015). \"Neuroinformatics Software Applications Supporting Electronic Data Capture, Management, and Sharing for the Neuroimaging Community\". Neuropsychology Review 25 (3): 356-68. doi:10.1007\/s11065-015-9293-x. PMC PMC5400666. PMID 26267019. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5400666 .   \n\n\u2191 Van Horn, J.D.; Toga, A.W. (2009). \"Is it time to re-prioritize neuroimaging databases and digital repositories?\". NeuroImage 47 (4): 1720-34. doi:10.1016\/j.neuroimage.2009.03.086. PMC PMC2754579. PMID 19371790. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2754579 .   \n\n\u2191 Marcus, D.S.; Harms, M.P.; Snyder, A.Z. et al. (2013). \"Human Connectome Project informatics: quality control, database services, and data visualization\". NeuroImage 80: 202-19. doi:10.1016\/j.neuroimage.2013.05.077. PMC PMC3845379. PMID 23707591. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3845379 .   \n\n\u2191 Das, S.; Zijdenbos, A.P.; Harlap, J. et al. (2012). \"LORIS: A web-based data management system for multi-center studies\". Frontiers in Neuroinformatics 5: 37. doi:10.3389\/fninf.2011.00037. PMC PMC3262165. PMID 22319489. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3262165 .   \n\n\u2191 Book, G.A.; Anderson, B.M.; Stevens, M.C. et al. (2013). \"Neuroinformatics Database (NiDB) - A modular, portable database for the storage, analysis, and sharing of neuroimaging data\". Neuroinformatics 11 (4): 495-505. doi:10.1007\/s12021-013-9194-1. PMC PMC3864015. PMID 23912507. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3864015 .   \n\n\u2191 \"OpenClinica User Documentation\". OpenClinica, LLC. 18 April 2016. https:\/\/docs.openclinica.com\/ .   \n\n\u2191 Harris, P.A.; Taylor, R.; Thielke, R. et al. (2009). \"Research electronic data capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support\". Journal of Biomedical Informatics 42 (2): 377\u201381. doi:10.1016\/j.jbi.2008.08.010. PMC PMC2700030. PMID 18929686. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2700030 .   \n\n\u2191 Krestyaninova, M.; Zarins, A.; Viksna, J. et al. (2009). \"A system for information management in biomedical studies \u2013 SIMBioMS\". Bioinformatics 25 (20): 2768-2769. doi:10.1093\/bioinformatics\/btp420. PMC PMC2759553. PMID 19633095. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2759553 .   \n\n\u2191 Scott, A.; Courtney, W.; Wood, D. et al. (2011). \"COINS: An Innovative Informatics and Neuroimaging Tool Suite Built for Large Heterogeneous Datasets\". Frontiers in Neuroinformatics 5: 33. doi:10.3389\/fninf.2011.00033. PMC PMC3250631. PMID 22275896. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3250631 .   \n\n\u2191 15.0 15.1 15.2 \"CubicWeb - The Semantic Web is a construction game!\". Logilab. 2016. https:\/\/www.cubicweb.org\/ .   \n\n\u2191 16.0 16.1 Schumann, G.; Loth, E.; Banaschewski, T. et al. (2010). \"The IMAGEN study: Reinforcement-related behaviour in normal brain function and psychopathology\". Molecular Psychiatry 15 (12): 1128-39. doi:10.1038\/mp.2010.4. PMID 21102431.   \n\n\u2191 17.0 17.1 Murphy, D.; Spooren, W. (2012). \"EU-AIMS: A boost to autism research\". Nature Reviews Drug Discovery 11 (11): 815-6. doi:10.1038\/nrd3881. PMID 23123927.   \n\n\u2191 Michel, V.; Schwartz, Y.; Pinel, P. et al. (2013). \"Brainomics: A management system for exploring and merging heterogeneous brain mapping data\". Proceedings from the 19th Annual Meeting of the Organization for Human Brain Mapping 2013. https:\/\/hal.inria.fr\/cea-00904768\/en .   \n\n\u2191 Papadopoulos Orfanos, D.; Michel, V.; Schwartz, Y. et al. (2017). \"The Brainomics\/Localizer database\". NeuroImage 144 (Pt B): 309-314. doi:10.1016\/j.neuroimage.2015.09.052. PMID 26455807.   \n\n\u2191 Prud'hommeaux, E.; Seaborne, A., ed. (15 January 2008). \"SPARQL Query Language for RDF\". World Wide Web Consortium. https:\/\/www.w3.org\/TR\/rdf-sparql-query\/ .   \n\n\u2191 Gorgolewski, K.; Burns, C.D.; Madison, C. et al. (2011). \"Nipype: A flexible, lightweight and extensible neuroimaging data processing framework in Python\". Frontiers in Neuroinformatics 5 (Pt B): 13. doi:10.3389\/fninf.2011.00013. PMC PMC3159964. PMID 21897815. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3159964 .   \n\n\u2191 Chapman, B.; Chang, J. (2000). \"Biopython: Python tools for computational biology\". ACM SIGBIO Newsletter 20 (2): 15\u201319. doi:10.1145\/360262.360268.   \n\n\u2191 Abraham, A.; Pedregosa, F.; Eickenberg, M. et al. (2014). \"Machine learning for neuroimaging with Scikit-learn\". Frontiers in Neuroinformatics 8: 14. doi:10.3389\/fninf.2014.00014. PMC PMC3930868. PMID 24600388. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3930868 .   \n\n\u2191 Fischer, C.; Operto, G.; Laguitton, S. et al.. \"Morphologist 2012: The new morphological pipeline of BrainVisa\". Proceedings from the Human Brain Mapping Conference 2012 2012.   \n\n\u2191 \"libfuse\/libfuse\". GitHub, Inc. https:\/\/github.com\/libfuse\/libfuse .   \n\n\u2191 Rockhold, F.; Bishop, S. (2012). \"Extracting the value of standards: The role of CDISC in a pharmaceutical research strategy\". Clinical Evaluation 40: 91\u201396.   \n\n\u2191 NSAp developers. \"Rql Upload\". GitHub, Inc. http:\/\/neurospin.github.io\/rql_upload\/ .   \n\n\u2191 28.0 28.1 NSAp developers. \"Population Imaging Web Service: PIWS\". GitHub, Inc. http:\/\/neurospin.github.io\/piws\/ .   \n\n\u2191 \"neurospin\/zeijemol\". GitHub, Inc. https:\/\/github.com\/neurospin\/zeijemol .   \n\n\u2191 NSAp developers. \"Rql Download\". GitHub, Inc. http:\/\/neurospin.github.io\/rql_download\/ .   \n\n\u2191 \"neurospin\/brainomics2\". GitHub, Inc. https:\/\/github.com\/neurospin\/brainomics2 .   \n\n\u2191 \"What is Twisted?\". Software Freedom Conservancy. https:\/\/twistedmatrix.com\/trac\/ .   \n\n\u2191 Schwartz, Y.; Barbot, A.; Thyreau, B. et al. (2012). \"PyXNAT: XNAT in Python\". Frontiers in Neuroinformatics 6: 12. doi:10.3389\/fninf.2012.00012. PMC PMC3354345. PMID 22654752. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3354345 .   \n\n\u2191 NSAp developers. \"CWBROWSER\". GitHub, Inc. http:\/\/neurospin.github.io\/rql_download\/cwbrowser .   \n\n\u2191 Keator, D.B.; Helmer, K.; Steffener, J. et al. (2013). \"Towards structured sharing of raw and derived neuroimaging data across existing resources\". Neuroimage 82: 647-61. doi:10.1016\/j.neuroimage.2013.05.094. PMC PMC4028152. PMID 23727024. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4028152 .   \n\n\u2191 Poldrack, R.A.; Kittur, A.; Kalar, D. et al. (2011). \"The cognitive atlas: Toward a knowledge foundation for cognitive neuroscience\". Frotiers in Neuroinformatics 5: 17. doi:10.3389\/fninf.2011.00017. PMC PMC3167196. PMID 21922006. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3167196 .   \n\n\u2191 Gibaud, B.; Kassel, G.; Dojat, M. et al. (2011). \"NeuroLOG: Sharing neuroimaging data using an ontology-based federated approach\". AMIA Annual Symposium Proceedings 2011: 472-80. PMC PMC3243145. PMID 22195101. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3243145 .   \n\n\u2191 Dumontier, M.; Callahan, A.; Cruz-Toledo, J. et al. (2014). \"Bio2RDF release 3: A larger connected network of linked data for the life sciences\". Proceedings of the 2014 International Conference on Posters & Demonstrations Track 1272: 401\u201304. http:\/\/ceur-ws.org\/Vol-1272\/paper_121.pdf .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. References are in order of appearance rather than alphabetical order (as the original was) due to the way this wiki works. Footnotes (URLs to projects) were turned into full citations. The URL to the zeijemol application cube was broken, and a direct GitHub URL was substituted instead.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\">https:\/\/www.limswiki.org\/index.php\/Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on neuroinformatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 20 June 2017, at 16:58.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 985 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","bce85c098ea6958c92b6dcce94e42565_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Neuroimaging_genetics_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Neuroimaging, genetics, and clinical data sharing in Python using the CubicWeb framework<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>In neurosciences or psychiatry, the emergence of large multi-center population imaging studies raises numerous technological challenges. From distributed data collection, across different institutions and countries, to final data publication service, one must handle the massive, heterogeneous, and complex data from genetics, imaging, demographics, or clinical scores. These data must be both efficiently obtained and downloadable. We present a Python solution, based on the CubicWeb open-source semantic framework, aimed at building population imaging study repositories. In addition, we focus on the tools developed around this framework to overcome the challenges associated with data sharing and collaborative requirements. We describe a set of three highly adaptive web services that transform the CubicWeb framework into a (1) multi-center upload platform, (2) collaborative quality assessment platform, and (3) publication platform endowed with massive-download capabilities. Two major European projects, IMAGEN and EU-AIMS, are currently supported by the described framework. We also present a Python package that enables end users to remotely query neuroimaging, genetics, and clinical data from scripts.\n<\/p><p><b>Keywords<\/b>: web service, data sharing, database, neuroimaging, genetics, medical informatics, Python\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Health research strategies using neuroimaging have shifted in recent years: the focus has moved from patient care only, to a combination of patient care and prevention. In the case of neurodegenerative and psychiatric diseases, this drives the creation of increasingly numerous massive imaging studies, also known as population imaging (PI) surveys.<sup id=\"rdp-ebb-cite_ref-HurkoTheADNI12_1-0\" class=\"reference\"><a href=\"#cite_note-HurkoTheADNI12-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PoldrackMaking14_2-0\" class=\"reference\"><a href=\"#cite_note-PoldrackMaking14-2\" rel=\"external_link\">[2]<\/a><\/sup> It should be noticed that PI studies no longer consist of image data only. The recent wide availability of high-throughput genomics has augmented the subject data with genetics, epigenetics, and functional genomics. Likewise, the standardization of personality, demographics, and deficit tests in psychiatry facilitates the acquisition of clinical\/behavioral records to enrich the subject data in large population studies. Moreover, PI studies now classically encompass more than one single imaging session per subject and cover multiple-time point heterogeneous experiments. Ultimately, these studies with complex imaging and extended data (PIx) require multi-center acquisitions to build a large target population.\n<\/p><p>A regular PIx infrastructure has to cover the following three main topics: (1) data collection, (2) quality control (QC) with data processing, and (3) data indexing and publication with controlled data sharing mechanisms. Furthermore, PIx infrastructures must evolve during the life cycle of a population imaging project, and they must also be resilient to extreme evolutions of the data content and management. In the projects we manage, we experience several extreme evolutions. The first kind of evolution may affect the published dataset such as adding a new modality for all subjects, a new time point or a new subcohort. Second, the amount of data requested evolves dramatically as the project consortium gets enlarged.<sup id=\"rdp-ebb-cite_ref-GorgolewskiFront15_3-0\" class=\"reference\"><a href=\"#cite_note-GorgolewskiFront15-3\" rel=\"external_link\">[3]<\/a><\/sup> Finally, internal ontologies have to evolve constantly in order to match the ongoing initiatives on interoperability.<sup id=\"rdp-ebb-cite_ref-Scheufele_tranSMART14_4-0\" class=\"reference\"><a href=\"#cite_note-Scheufele_tranSMART14-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GorgolewskiTheBrain16_5-0\" class=\"reference\"><a href=\"#cite_note-GorgolewskiTheBrain16-5\" rel=\"external_link\">[5]<\/a><\/sup>\n<\/p><p>Several existing open-source frameworks support one or several of the described topics, sometimes only for one specific data type. We propose in the following a brief overview of existing systems. Some of these systems have also been reviewed by Nichols and Pohl.<sup id=\"rdp-ebb-cite_ref-NicholsNeuro15_6-0\" class=\"reference\"><a href=\"#cite_note-NicholsNeuro15-6\" rel=\"external_link\">[6]<\/a><\/sup> IDA<sup id=\"rdp-ebb-cite_ref-HornIsIt09_7-0\" class=\"reference\"><a href=\"#cite_note-HornIsIt09-7\" rel=\"external_link\">[7]<\/a><\/sup> is a neuroimaging data repository and management system that supports data collection (topic one) and data sharing (topic three). With this system, the published datasets can be searched using automatically extracted metadata. The XNAT framework<sup id=\"rdp-ebb-cite_ref-MarcusHuman13_8-0\" class=\"reference\"><a href=\"#cite_note-MarcusHuman13-8\" rel=\"external_link\">[8]<\/a><\/sup> is widely used for neuroimaging data and supports all the PIx infrastructure topics, focusing on tools to pipeline, and to audit the processing of image data (topic two). The LORIS<sup id=\"rdp-ebb-cite_ref-DasLORIS12_9-0\" class=\"reference\"><a href=\"#cite_note-DasLORIS12-9\" rel=\"external_link\">[9]<\/a><\/sup> and NiDB<sup id=\"rdp-ebb-cite_ref-BookNeuroinfo13_10-0\" class=\"reference\"><a href=\"#cite_note-BookNeuroinfo13-10\" rel=\"external_link\">[10]<\/a><\/sup> frameworks represent a significant effort to account for multimodal data involved in PIx studies. These frameworks, although addressing all the required topics, mainly support neuroimaging data. Openclinica<sup id=\"rdp-ebb-cite_ref-OpenClinicaUser16_11-0\" class=\"reference\"><a href=\"#cite_note-OpenClinicaUser16-11\" rel=\"external_link\">[11]<\/a><\/sup> and REDCap<sup id=\"rdp-ebb-cite_ref-HarrisResearch09_12-0\" class=\"reference\"><a href=\"#cite_note-HarrisResearch09-12\" rel=\"external_link\">[12]<\/a><\/sup> facilitate the collection of electronic data such as eCRF or questionnaires and are recognized in projects of various sizes that support data collection (topic one). Likewise, <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> were developed for the collection of genomic measurements such as SIMBioMS.<sup id=\"rdp-ebb-cite_ref-KrestyaninovaASys09_13-0\" class=\"reference\"><a href=\"#cite_note-KrestyaninovaASys09-13\" rel=\"external_link\">[13]<\/a><\/sup> Finally, the COINS framework brings essential tools for multimodal data support and, more interestingly, emphasizes the importance of providing sharing tools (topics one and three).<sup id=\"rdp-ebb-cite_ref-ScottCOINS11_14-0\" class=\"reference\"><a href=\"#cite_note-ScottCOINS11-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/p><p>The two European studies we manage require a tailored PIx infrastructure. Existing frameworks neither completely handle the diversity of our PIx requirements and project life cycle nor provide efficient tools to collect, check the quality of, and publish evolving data. Additional developments were required for building such complete infrastructure. We based these developments on a more general framework than the dedicated applications described above. In collaboration with Logilab company (Logilab SA, Paris, France), we developed three highly adaptive web services, based on the CubicWeb (CW) pure-Python framework, aimed at creating a (1) multi-center upload platform, (2) collaborative quality assessment platform, and (3) publication platform with massive-download features.<sup id=\"rdp-ebb-cite_ref-LogilabCubicWeb_15-0\" class=\"reference\"><a href=\"#cite_note-LogilabCubicWeb-15\" rel=\"external_link\">[15]<\/a><\/sup> These developments were originally instituted for IMAGEN and EU-AIMS projects in order to host their data about mental health in adolescents<sup id=\"rdp-ebb-cite_ref-SchumannTheIMAGEN10_16-0\" class=\"reference\"><a href=\"#cite_note-SchumannTheIMAGEN10-16\" rel=\"external_link\">[16]<\/a><\/sup> and autism<sup id=\"rdp-ebb-cite_ref-MurphyEU-AIMS12_17-0\" class=\"reference\"><a href=\"#cite_note-MurphyEU-AIMS12-17\" rel=\"external_link\">[17]<\/a><\/sup>, respectively. The corresponding studies require key features such as upload\/browse published data from the web, dynamic selection and filtering of displayed data, support for flexible download operations, high-level request language, multilevel access rights, remote data access, remote user access rights management, collaborative QC, and interoperability.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<p>The three services described in the introduction were handled in distinct developments. The next sub-section presents the CW framework capabilities, followed by introductions in the second and fourth sub-sections to the upload and publication web services through which the tailored requirements of PIx studies are satisfied. Furthermore, we describe in the third sub-section a collaborative rating web service that helps users to assess the data quality, and in the final sub-section a Python API that remotely queries these web services.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"CubicWeb_overview\">CubicWeb overview<\/span><\/h3>\n<p>All the implemented services are based on the CW framework.<sup id=\"rdp-ebb-cite_ref-LogilabCubicWeb_15-1\" class=\"reference\"><a href=\"#cite_note-LogilabCubicWeb-15\" rel=\"external_link\">[15]<\/a><\/sup> We choose a high level pure-Python framework that bridges web technologies and database engines. This choice was also based on the expertise and experience of people from our <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> and a tight collaboration with Logilab.<sup id=\"rdp-ebb-cite_ref-MichelBrainomics13_18-0\" class=\"reference\"><a href=\"#cite_note-MichelBrainomics13-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PapadopoulosTheBrainomics17_19-0\" class=\"reference\"><a href=\"#cite_note-PapadopoulosTheBrainomics17-19\" rel=\"external_link\">[19]<\/a><\/sup> CW distribution is organized in a core part and a set of basic Python modules, referred to as cubes, which can be used to efficiently generate web applications. The core of the CW framework, developed under the LGPL license, is constructed from well-established technologies (SQL, Python, web technologies such as HTML5 and Javascript). The main characteristics of the CW framework are given as follows:\n<\/p>\n<dl><dd>1. CW defines its data model with Python classes and automatically generates the underlying database structure.<\/dd><\/dl>\n<dl><dd>2. The queries are expressed with the RQL language, which is similar to W3C\u2019s SPARQL.<sup id=\"rdp-ebb-cite_ref-Prud.27hommeauxSPARQL08_20-0\" class=\"reference\"><a href=\"#cite_note-Prud.27hommeauxSPARQL08-20\" rel=\"external_link\">[20]<\/a><\/sup> All the persistent data are retrieved and modified using this language.<\/dd><\/dl>\n<dl><dd>3. CW implements a mechanism that exposes <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> in several ways, referred to as views. This mechanism implements the classical model-view-controller software architecture pattern. Defined in Python, the views are applied to query results and can produce HTML pages and\/or trigger external processes. The separation of queries and views offers major advantages: first, the same data selection may have several web representations, and second, retrieved data can be exported in several other formats without modifying the underlying data storage.<\/dd><\/dl>\n<dl><dd>4. All the views and triggers are recorded in a registry and are automatically selected depending on the current context, which is inferred from the type of data returned by the RQL.<\/dd><\/dl>\n<dl><dd>5. Thanks to the semantic nature of CW, all developments inherit the possibility to follow existing or emerging ontologies, thereby facilitating sharing, access, and processing.<\/dd><\/dl>\n<dl><dd>6. CW has a security system that grants fine-grained access to the data. This system is similar to the row-level security and policies available in the most recent versions of <a href=\"https:\/\/www.limswiki.org\/index.php\/PostgreSQL\" title=\"PostgreSQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"a5dd945cdcb63e2d8f7a5edb3a896d82\">PostgreSQL<\/a>, and links access rights to entities\/relations in the schema. Each entity type has a set of attributes and relations, and permissions that define who can add, read, update, or delete such an entity and associated relations.<\/dd><\/dl>\n<dl><dd>7. CW may run either as a standalone application or behind an Apache front server. We refer to both settings as a data sharing service (DSS) (cf. Figure 1).<\/dd><\/dl>\n<dl><dd>8. CW can be configured to run with various database engines. For the best performance, PostgreSQL is recommended.<\/dd><\/dl>\n<p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Grigis_FInNeuroinformatics2017_11.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"511bbebcfe35ad61ecff2048ecc63b64\"><img alt=\"Fig1 Grigis FInNeuroinformatics2017 11.jpg\" src=\"https:\/\/www.limswiki.org\/images\/6\/63\/Fig1_Grigis_FInNeuroinformatics2017_11.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Architecture of a CubicWeb data sharing service (DSS) integrated in an Apache platform with LDAP. The business logic cubes provide a schema that can be instantiated in the database management system (DBMS: red puzzle piece). The system cubes ensure low-level system interactions (green puzzle piece), and the application cube proposes a web user interface (blue puzzle piece). End users access the database content through a web browser, a Python API scripting the DSS or an FTP solution, where virtual folders (acting as filters on the central repository) are proposed for download.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Starting from the basic CW distributions, our suite of services is composed of an assembly of Python modules, also referred to as cubes. The Python language is widely used in scientific communities and facilitates interfacing with major or emerging processing tools such as Nipype<sup id=\"rdp-ebb-cite_ref-GorgolewskiNipype11_21-0\" class=\"reference\"><a href=\"#cite_note-GorgolewskiNipype11-21\" rel=\"external_link\">[21]<\/a><\/sup>, Biopython<sup id=\"rdp-ebb-cite_ref-ChapmanBiopython00_22-0\" class=\"reference\"><a href=\"#cite_note-ChapmanBiopython00-22\" rel=\"external_link\">[22]<\/a><\/sup>, Nilearn<sup id=\"rdp-ebb-cite_ref-AbrahamMachine14_23-0\" class=\"reference\"><a href=\"#cite_note-AbrahamMachine14-23\" rel=\"external_link\">[23]<\/a><\/sup>, and Morphologist.<sup id=\"rdp-ebb-cite_ref-FischerMorph12_24-0\" class=\"reference\"><a href=\"#cite_note-FischerMorph12-24\" rel=\"external_link\">[24]<\/a><\/sup> Application cubes, built over system cubes, and business logic cubes can be distinguished. The system cubes ensure interactions with the operating system and middleware. For example, they connect to LDAP for user credentials and information or invoke FUSE<sup id=\"rdp-ebb-cite_ref-GitHubFUSE_25-0\" class=\"reference\"><a href=\"#cite_note-GitHubFUSE-25\" rel=\"external_link\">[25]<\/a><\/sup> as a module to construct virtual file systems in a user repository for downloading. The business logic cubes essentially provide the database schema and the application cubes define the access rights and the web interface.\n<\/p><p>Among the available Python-based frameworks, we chose CW. A major advantage of CW is the RQL language which brings end users a query interface adapted for PIx data sharing. It simplifies and improves the user experience in searching for custom datasets. RQL also avoids the use of a complicated object relational mapper (ORM), is focused on browsing relations, and allows requesting several DSS at once. The semantic nature of this request language requires the user to know only about the used data model defined as a graph (nothing about the underlying low-level relational model). This data model simplification and the expressiveness of RQL help users writing custom requests, while most of existing DSS do not expose a query language but offer a limited predefined number of operations that can be carefully designed to be efficient (e.g., RESTful APIs). Criticisms against systems exposing a query request language to the end users emphasize a risk of denial of service. To avoid this issue (i.e., overloading the server with arbitrary complex requests), CW allows limitation of usable resources (RAM per request, CPU per request, number of requests per user, CPU time per request). We believe that users should be able to select and download only what they specifically need using a query request language. This avoids filtering the data locally and saves the bandwidth.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Structured_data_upload_service\">Structured data upload service<\/span><\/h3>\n<p>In PIx studies, massive and complex data are gathered from multiple data acquisition centers or devices (topic one). Each collected dataset must be mapped with definitions that follow consensus representation rules. Those definitions are grouped in data dictionaries that ideally follow standards<sup id=\"rdp-ebb-cite_ref-RockholdExtract12_26-0\" class=\"reference\"><a href=\"#cite_note-RockholdExtract12-26\" rel=\"external_link\">[26]<\/a><\/sup>, but they are mainly manufacturer- and\/or site-specific. Thus, an efficient and versatile tool is required for mapping the different data dictionaries during the collection process.\n<\/p><p>Leveraging those ideas, we implemented a flexible upload mechanism, a system cube named <i>rql_upload<\/i><sup id=\"rdp-ebb-cite_ref-NSApRQLUpload_27-0\" class=\"reference\"><a href=\"#cite_note-NSApRQLUpload-27\" rel=\"external_link\">[27]<\/a><\/sup> and provided a web frontend by integrating this cube with the application cube named <i>PIWS<\/i><sup id=\"rdp-ebb-cite_ref-NSApPIWS_28-0\" class=\"reference\"><a href=\"#cite_note-NSApPIWS-28\" rel=\"external_link\">[28]<\/a><\/sup> (Population Imaging Web Service, cf. Figure 1). Based on a CW feature that allows database completion through online HTML forms, these two cubes were developed to collect, in a DSS, both raw data and metadata. CW also enables the customization of triggers that determine the integrity of the uploaded data: synchronous and asynchronous validation filters can be specified and applied to each upload dataset. The upload proceeds as follows (cf. Figure 2):\n<\/p>\n<dl><dd>1. Synchronous validations are applied to each form field (e.g., to check the extension of a file or the structure of an Excel table). If the validation filtering fails, then the web form is refreshed and an adapted feedback is displayed.<\/dd><\/dl>\n<dl><dd>2. After synchronous validation, all the uploaded raw data\/metadata are stored in generic entities and a \"Quarantine\" status is set. To avoid cluttering of the database and to ease file manipulation, files are stored in the central repository but remain accessible through the database. File hashes are automatically computed and indexed in order to assess data integrity.<\/dd><\/dl>\n<dl><dd>3. To update the upload status from \"Quarantine\" to \"Rejected\"\/\"Validated,\" automatic asynchronous validations can be configured in the service as looping tasks. Those validation filters are project and\/or data and\/or upload specific and generate adapted feedback for users and data managers.<\/dd><\/dl>\n<p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Grigis_FInNeuroinformatics2017_11.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"b2be30737501c6529bfc251cc9e19582\"><img alt=\"Fig2 Grigis FInNeuroinformatics2017 11.jpg\" src=\"https:\/\/www.limswiki.org\/images\/3\/33\/Fig2_Grigis_FInNeuroinformatics2017_11.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Illustration of the upload process. The <b>(A)<\/b> syntax of a form description JSON file, <b>(B)<\/b> corresponding web form as presented to users (here an error message returned by synchronous validation is displayed in the top red box), <b>(C)<\/b> \u201cQuarantine\u201d status, and <b>(D)<\/b> \u201cValidated\u201d status (obtained after asynchronous validation) as displayed to users: note that no feedback is shown here.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Moreover, any entity or relation may be endowed with access permission rules.<sup id=\"rdp-ebb-cite_ref-LogilabCubicWeb_15-2\" class=\"reference\"><a href=\"#cite_note-LogilabCubicWeb-15\" rel=\"external_link\">[15]<\/a><\/sup> Based on the CW security mechanisms, a customized security model was implemented for our upload DSS (it can be extended later). Only specific groups have the authorization to upload, and users can only access the uploads, which they are interested in. The customization of these core features allowed the creation of an upload web service that is completely described in a single JSON file. This file links the web form fields with customized or CW-internal controllers that manage the type of data to be collected.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Collaborative_quality_control_service\">Collaborative quality control service<\/span><\/h3>\n<p>Owing to the large amount of data gathered\/analyzed in PIx studies, we must consider more sophisticated operating procedures than simple quality controls (QCs), where datasets are usually only rated once by a handful of individuals. This issue can be addressed by implementing a web-based collaborative quality control process that will also remove the bias introduced by isolated raters (topic two). Moreover, for the studies we manage, we also added controlled vocabulary description to the ratings.\n<\/p><p>We achieve these goals by implementing a flexible collaborative rating mechanism, i.e., an application cube named <i>zeijemol<\/i>.<sup id=\"rdp-ebb-cite_ref-GitHubZeijemol_29-0\" class=\"reference\"><a href=\"#cite_note-GitHubZeijemol-29\" rel=\"external_link\">[29]<\/a><\/sup> As in the previous sub-section, a collaborative quality control DSS is entirely described in a single JSON file. This file consists, on the one hand, of the list of elements that will be rated (e.g., a Nifti image, a FreeSurfer segmentation, or a motion curve in a diffusion sequence of an individual) and, on the other hand, related quality indicators (e.g., binary good\/bad, controlled vocabulary, scaled rating). Each element is displayed by one of the embedded viewers such as triplanar view or mesh rendering (cf. Figure 3). The QC results are stored directly in the database.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Grigis_FInNeuroinformatics2017_11.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"236ad6d64aecacbf31760b5244521230\"><img alt=\"Fig3 Grigis FInNeuroinformatics2017 11.jpg\" src=\"https:\/\/www.limswiki.org\/images\/6\/6a\/Fig3_Grigis_FInNeuroinformatics2017_11.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> The collaborative quality control web service of a FreeSurfer segmentation element of one subject. <b>(A)<\/b> the quality indicators (in this case, a controlled vocabulary with an accept\/prescribe manual edit\/reject decision and an optional check-box justification), <b>(B)<\/b> a triplanar view of the white and pial surfaces overlayed on the anatomical image, and <b>(C)<\/b> the white and pial meshes with statistical indicators.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The emergence of such DSS will allow machine learning techniques to learn new classifiers to automatized the quality control task. The QC scores may also be directly used as prior knowledge during the analysis stage.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Publication_service\">Publication service<\/span><\/h3>\n<p>In PIx studies, data collection and QC are followed by data anonymization, ordering, and analysis. Ultimately, data are made available to the acquisition partners or the scientific community (topic three). While browsing the database content through the web interface, users expect to be able to download the displayed files as well as the data description and rich links between the data, also referred to as metadata. An intuitive and reliable sharing mechanism is therefore crucial as large amounts of heterogeneous evolving data must be provided. Furthermore, for the studies we manage, access rights are split along time points, scan types, questionnaires, or questions to match the consortia multilevel access permissions.\n<\/p><p>Therefore, we implemented a system cube named <i>rql_download<\/i><sup id=\"rdp-ebb-cite_ref-NSApRQLDownload_30-0\" class=\"reference\"><a href=\"#cite_note-NSApRQLDownload-30\" rel=\"external_link\">[30]<\/a><\/sup> and provided a web frontend by integrating this cube with <i>PIWS<\/i><sup id=\"rdp-ebb-cite_ref-NSApPIWS_28-1\" class=\"reference\"><a href=\"#cite_note-NSApPIWS-28\" rel=\"external_link\">[28]<\/a><\/sup> whenever it was used in a publication service (cf. Figure 1). The <i>rql_download<\/i> cube converts the result of any RQL query into files on a virtual file system that, in turn, can be accessed through a secured file transfer protocol (sFTP) (cf. Figure 1). The following five sub-sections introduce: (1) the business logic cubes used to describe the neuroimaging genetics data and metadata and the relationship between these data; (2) how users can save the content of their current search from the DSS web interface; (3) two approaches of <i>rql_download<\/i>, based on two basic softwares (FUSE or Twisted), that give users access to their saved searches, as well as the pros and cons of both; (4) a suitable strategy for setting user rights from the CW security system; and (5) a descriptive data insertion mechanism, as a set of Python scripts.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"A_dedicated_structure_for_imaging_genomics_questionnaire_data\">A dedicated structure for imaging genomics questionnaire data<\/span><\/h4>\n<p>The database schema was developed for handling multi-time point\/multimodal datasets in the brainomics business logic cube.<sup id=\"rdp-ebb-cite_ref-GitHubBrainomics2_31-0\" class=\"reference\"><a href=\"#cite_note-GitHubBrainomics2-31\" rel=\"external_link\">[31]<\/a><\/sup> This schema supports general information such as subject data and associated metadata (age, handedness, sex, etc.), acquisition center definitions, multimodal imaging datasets, clinical\/behavioral records, processed data, and some genomic concepts (including chromosomes, genes, SNPs, or genomic platforms). An excerpt of the produced schema is shown in Figure 4.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Grigis_FInNeuroinformatics2017_11.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"33d7be14b88c3684c75160ccc1a35bcb\"><img alt=\"Fig4 Grigis FInNeuroinformatics2017 11.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/fa\/Fig4_Grigis_FInNeuroinformatics2017_11.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> A snippet of the schema used in a publication DSS. We see from the green boxes that all entities are related to an \u201cAssessment\u201d entity through an \u201cin_assessment\u201d relation. This behavior is inherited from the access rights described in the fourth sub-section.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"Efficient_data_selection_and_download_tool:_The_data_shopping_cart_mechanism\">Efficient data selection and download tool: The data shopping cart mechanism<\/span><\/h4>\n<p>When an RQL query result set is returned by the DSS, the most adapted view is automatically selected, and facets are attached to each web page, thereby providing filtering rules. Facets allow interactive and graphical search refinements in accordance with selected attributes (e.g., sex or handedness filter for a subject result set). The developed shopping cart mechanism serves to save the user searches that consist of data, possibly large files, and metadata. This mechanism and the facet filtering are smoothly integrated: activating a filter option from the web interface automatically updates the search query result set, and thus, the list of files that will be dropped for download (cf. Figure 5). The data added to one cart has an expiration date that can be configured in the service. Convenient access rights are set: users can only access their own searches. For the sake of the EU-AIMS project hosted in our laboratory, a video explaining the data shopping cart mechanism <a rel=\"external_link\" class=\"external text\" href=\"ftp:\/\/ftp.cea.fr\/pub\/unati\/euaims\/download_euaims_data.mp4\" target=\"_blank\">is available<\/a> (.mp4 file).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Grigis_FInNeuroinformatics2017_11.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"b7f5c486acacecb0374be5171038f9eb\"><img alt=\"Fig5 Grigis FInNeuroinformatics2017 11.jpg\" src=\"https:\/\/www.limswiki.org\/images\/1\/1b\/Fig5_Grigis_FInNeuroinformatics2017_11.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Illustration of the download process via the proposed shopping cart mechanism. <b>(A)<\/b> the facet filter bar when all the scans (\u201cScan\u201d entities) are requested (as highlighted in bold, the user has selected only the \u201cFU2\u201d time point and the diffusion MRI \u201cDTI\u201d scans), <b>(B)<\/b> the view corresponding to the filtered dataset, <b>(C)<\/b> add this new search to the cart (by activating these filtering options, the save RQL path search will be automatically updated), <b>(D)<\/b> a new search has been created, and <b>(E)<\/b> the download of the search and associated files as presented in FileZilla.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"The_transfer_of_the_shopping_cart_content:_Data_download\">The transfer of the shopping cart content: Data download<\/span><\/h4>\n<p>When saved, the cart content is made available as virtual files and folders. A major advantage of the developed solution is that data compression or duplication is avoided, and that in turn requires no extra load for the publication DSS. Data download operations are delegated to sFTP servers to ensure secure transfers. The sFTP is standard and supported by numerous client software on most systems.\n<\/p><p>Two approaches are implemented in the <i>rql_download<\/i> cube that can be selected by configuration settings:\n<\/p>\n<dl><dd>1. FUSE virtual folders: For each search, the system builds a list of files to be downloaded and subsequently creates a virtual FUSE directory acting as a filter on the central repository. The user can only see subsets of files\/directories corresponding to his queries built in accordance with his access rights. Finally, the system delegates the data transfers to the sFTP server. The major advantage of this approach is the use of the standard sFTP port. However, additional system level configurations are required during the installation of the DSS in order to set the user home directories and system accounts.<\/dd><\/dl>\n<dl><dd>2. Twisted server: This approach is characterized by a Python process that creates a Twisted<sup id=\"rdp-ebb-cite_ref-SFCTwisted_32-0\" class=\"reference\"><a href=\"#cite_note-SFCTwisted-32\" rel=\"external_link\">[32]<\/a><\/sup> event-driven networking server, retrieves all the searches in the database, and exposes the files via sFTP through the created server. Again, this process acts as a filter on the central repository where a user only sees a subset of files\/directories. In this case, the authentication and file transfers are directly operated by CW. The major advantage of this strategy is that no system-level configuration is required. However, listening on a non-default sFTP port, which could lead to firewall issues, is sometimes required.<\/dd><\/dl>\n<h4><span class=\"mw-headline\" id=\"Access_rights_mechanism\">Access rights mechanism<\/span><\/h4>\n<p>In the CW security model, any entity or relation may be endowed with permission rules. To fulfill consortia's criteria, we propose an operational setup of the CW security model for our publication DSS. We built our security model around \"pivotal entities\" rather than specifying rights on all entities. Pivotal entities are those on which access rights are defined, and they are related to all entities that must be covered by the security model through a specific relation (the \"in_assessment\" relation in Figure 4). Each time an entity covered by the security model is requested, the system automatically requests its related pivotal entity and propagates the corresponding access rights.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"The_unified_insertion_procedure\">The unified insertion procedure<\/span><\/h4>\n<p>A unified insertion module is provided as a set of Python scripts to insert neuroimaging, genomic, and clinical data such as scans, genomic measures, questionnaires, and processing steps. These scripts were helpful in efficiently managing the large amount of evolving data in our projects. The indexed data are uniformly organized according to the schema structure and thus take advantage of all the aforementioned developments (e.g., shopping cart mechanism and security model of previous sub-sections, and common renderings cf. Figure 6). Generating such a DSS with these scripts can be performed without specific CW knowledge. Indeed, only a rich description of the data to be published is required as a set of Python dictionary objects.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Grigis_FInNeuroinformatics2017_11.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"984b5bb8b1124582d39ff63825aafddc\"><img alt=\"Fig6 Grigis FInNeuroinformatics2017 11.jpg\" src=\"https:\/\/www.limswiki.org\/images\/c\/c9\/Fig6_Grigis_FInNeuroinformatics2017_11.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> Summary views of the database status. Global information, for example the <b>(A)<\/b> gender or <b>(B)<\/b> handedness distributions, <b>(C)<\/b> acquisition status, and <b>(D)<\/b> age distribution, or longitudinal information, such as <b>(E)<\/b> the answers of subject2 to specific questions across the study time points.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"A_transverse_Python_module_to_remotely_connect_a_CubicWeb_DSS\">A transverse Python module to remotely connect a CubicWeb DSS<\/span><\/h3>\n<p>With the aforementioned capabilities of the DSS, a user manually selects and downloads data through graphical interfaces in order to analyze them locally (as discussed in the previous sub-section). In the case of an evolving DSS, the downloaded data must be regularly updated, and this manual process becomes time consuming and error prone when large and heterogeneous data are considered. Moreover, the metadata, such as quality scores, used to specify the dataset to download are also likely to change. Therefore, to achieve the analysis of up-to-date data stored in a DSS, direct programmatic interaction with the DSS is recommended. In the neuroimaging and neuroscience communities, data are typically analyzed by using Python scripts. Classically, the systems provide RESTful web services such as XNAT, with a Python API.<sup id=\"rdp-ebb-cite_ref-SchwartzPyXNAT12_33-0\" class=\"reference\"><a href=\"#cite_note-SchwartzPyXNAT12-33\" rel=\"external_link\">[33]<\/a><\/sup> Inheriting from the RQL request language, our publication DSS (cf. previous sub-section) offers a rich interface to access the data.\n<\/p><p>We provide a regular Python module, named <i>cwbrowser<\/i><sup id=\"rdp-ebb-cite_ref-NSApCWBROWSER_34-0\" class=\"reference\"><a href=\"#cite_note-NSApCWBROWSER-34\" rel=\"external_link\">[34]<\/a><\/sup>, that implements a Python API to connect and send RQL to a remote DSS based on the CW framework. This module is completely independent of CW (no CW installation required) and similar to the CW distribution <i>cwclientlib<\/i> cube. A publication DSS, as described in the previous sub-section, can be requested by the <i>cwbrowser<\/i> module that embeds the previously described data selection and shopping cart capabilities. It automatically fills and saves a shopping cart from a custom RQL request, downloads the associated virtual directories onto the local file system, and returns the complete requested dataset. The returned dataset contains metadata stored in the DSS such as subject sex or quality scores as well as the path to the downloaded directories. These resources are organized following the DSS layout of files and folders. The users will get the same local tree which will help in writing sharable analysis scripts.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>Our laboratory operates several DSS for the IMAGEN project about mental health in adolescents<sup id=\"rdp-ebb-cite_ref-SchumannTheIMAGEN10_16-1\" class=\"reference\"><a href=\"#cite_note-SchumannTheIMAGEN10-16\" rel=\"external_link\">[16]<\/a><\/sup> and the EU-AIMS project about autism.<sup id=\"rdp-ebb-cite_ref-MurphyEU-AIMS12_17-1\" class=\"reference\"><a href=\"#cite_note-MurphyEU-AIMS12-17\" rel=\"external_link\">[17]<\/a><\/sup> Other DSS are currently under development to support new and ongoing initiatives. Note that the access to both IMAGEN and EU-AIMS datasets is (to date) restricted.\n<\/p><p>In the IMAGEN project, 2,000 subjects are monitored on at least two visits (the third follow-up is underway). T1, T2, FLAIR, DWI, B0, task fMRI, and resting-state fMRI scans are acquired, as well as clinical\/behavioral records, genotyping, gene expression, and methylation. A publication DSS at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/imagen2.cea.fr\/database\" target=\"_blank\">https:\/\/imagen2.cea.fr\/database<\/a> enables us to share more than 37,000 scans, 32,000 processing results, and 16 million distinct variables. In the near future, an upload DSS will allow us to collect a new time point.\n<\/p><p>In the EU-AIMS project, 1,500 subjects (from six months to 30 years old) are monitored on several visits through two distinct studies. T1, T2, FLAIR, DWI, B0, task fMRI, resting-state fMRI, and spectroscopy scans are acquired, as well as clinical\/behavioral records, EEG, eye-tracking, gene expression, and methylation. An upload DSS at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eu-aims.cea.fr\/database\" target=\"_blank\">https:\/\/eu-aims.cea.fr\/database<\/a> provides the means for collecting this data from 10 centers across Europe. In addition, a collaborative quality check DSS at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eu-aims.cea.fr\/qc\" target=\"_blank\">https:\/\/eu-aims.cea.fr\/qc<\/a> allows us to assess the uploaded data quality, and a publication DSS at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eu-aims.cea.fr\/data_repository\" target=\"_blank\">https:\/\/eu-aims.cea.fr\/data_repository<\/a> enables us to share more than 13,000 scans, 12,000 processing results, and 15 million distinct variables.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion_and_conclusion\">Discussion and conclusion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Lightweight_solution_for_data_sharing\">Lightweight solution for data sharing<\/span><\/h3>\n<p>We developed a novel and lightweight PIx software infrastructure exclusively based on the CW framework. We offer a suite of CW tools that facilitates the creation of a DSS. The system delivers the data to users based on the principle of \"what you see is what you get\": users define their datasets of interest by browsing the database. Thanks to the RQL and the developed Python API, remote query of a DSS is easy and intuitive. In this environment, core features such as the schema definition, the web rendering of the database content, and the semantic request language are provided by a few lines of Python code at the heart of the CW framework. Our DSS can use any database engine, offers an access permission mechanism, and can be smoothly integrated with the standard Apache environment. Moreover the CW framework relies on a large community of developers led by Logilab.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"A_PIx_Swiss_knife\">A PIx Swiss knife<\/span><\/h3>\n<p>CW is well suited for all the scenarios one can face in a PIx project. For instance, in the projects we manage, we also provided a CW-based service to allow a collaborative moderation of user access to the different DSS. This service enables the consortium review boards to assign the relevant access rights to new or existing users. It is restricted to a few members and enables the user account administration of an upload, collaborative QC, and publication DSS.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Future_directions\">Future directions<\/span><\/h3>\n<p>Our developments inherit the web semantic capabilities embedded in the CW framework. Thanks to this key feature, numerous problems of interoperability can be efficiently tackled using emerging ontologies and standards in <a href=\"https:\/\/www.limswiki.org\/index.php\/Neuroinformatics\" title=\"Neuroinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"4944aa70b984d0a743d4e9de46416311\">neuroinformatics<\/a>, neurosciences, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a>, such as the NIDM standard for data exchange<sup id=\"rdp-ebb-cite_ref-KeatorTowards13_35-0\" class=\"reference\"><a href=\"#cite_note-KeatorTowards13-35\" rel=\"external_link\">[35]<\/a><\/sup>, the Cognitive Atlas Ontology<sup id=\"rdp-ebb-cite_ref-PoldrackTheCog11_36-0\" class=\"reference\"><a href=\"#cite_note-PoldrackTheCog11-36\" rel=\"external_link\">[36]<\/a><\/sup>, and OntoNeuroLOG<sup id=\"rdp-ebb-cite_ref-GibaudNeuroLOG11_37-0\" class=\"reference\"><a href=\"#cite_note-GibaudNeuroLOG11-37\" rel=\"external_link\">[37]<\/a><\/sup> for data annotation, or the Bio2RDF for the federation of large datasets using open-source semantic web technologies.<sup id=\"rdp-ebb-cite_ref-DumontierBio2RDF14_38-0\" class=\"reference\"><a href=\"#cite_note-DumontierBio2RDF14-38\" rel=\"external_link\">[38]<\/a><\/sup> The annotation of our datasets, with respect to these ontologies, is ongoing. Ultimately, should all DSS follow standard ontologies, RQL would provide new cross-project querying possibilities. Although the CW framework is already used successfully in several commercial applications, it would be interesting to evaluate the CW framework performances on our DSS with Logilab dedicated tools.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h2>\n<p>AG developed the cubes, performed its deployment, and maintained the online repositories. DG, DO, TG, and RC tested the proposed application and used it in two European projects (IMAGEN, EU-AIMS). NC and AM developed the CubicWeb framework. VF, GS, WS, and DM initiated and supervized the projects. All authors contributed to the manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflict_of_interest_statement\">Conflict of interest statement<\/span><\/h2>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgments\">Acknowledgments<\/span><\/h2>\n<p>The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant agreement no. 115300 (EU-AIMS), resources of which are composed of financial contribution from the European Union\u2019s Seventh Framework Programme (FP7\/2007-2013) and EFPIA companies\u2019 in kind contribution.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_material\">Supplementary material<\/span><\/h2>\n<p>Codes are distributed under the terms of the CeCILL-B license, as published by the CEA-CNRS-INRIA. Refer to the license file or to <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cecill.info\/licences\/Licence_CeCILL-B_V1-en.html\" target=\"_blank\">http:\/\/www.cecill.info\/licences\/Licence_CeCILL-B_V1-en.html<\/a> for details. Codes are freely accessible on github <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/neurospin\" target=\"_blank\">https:\/\/github.com\/neurospin<\/a>. The DSS we are in charge of can be reached at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/imagen2.cea.fr\" target=\"_blank\">https:\/\/imagen2.cea.fr<\/a> and <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eu-aims.cea.fr\" target=\"_blank\">https:\/\/eu-aims.cea.fr<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HurkoTheADNI12-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HurkoTheADNI12_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hurko, O.; Black, S.E.; Doody, R. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3676932\" target=\"_blank\">\"The ADNI Publication Policy: Commensurate recognition of critical contributors who are not authors\"<\/a>. <i>NeuroImage<\/i> <b>59<\/b> (4): 4196\u20134200. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.neuroimage.2011.10.085\" target=\"_blank\">10.1016\/j.neuroimage.2011.10.085<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3676932\/\" target=\"_blank\">PMC3676932<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22100665\" target=\"_blank\">22100665<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3676932\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3676932<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+ADNI+Publication+Policy%3A+Commensurate+recognition+of+critical+contributors+who+are+not+authors&rft.jtitle=NeuroImage&rft.aulast=Hurko%2C+O.%3B+Black%2C+S.E.%3B+Doody%2C+R.+et+al.&rft.au=Hurko%2C+O.%3B+Black%2C+S.E.%3B+Doody%2C+R.+et+al.&rft.date=2012&rft.volume=59&rft.issue=4&rft.pages=4196%E2%80%934200&rft_id=info:doi\/10.1016%2Fj.neuroimage.2011.10.085&rft_id=info:pmc\/PMC3676932&rft_id=info:pmid\/22100665&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3676932&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PoldrackMaking14-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PoldrackMaking14_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Poldrack, R.A.; Gorgolewski, K.J. (2014). \"Making big data open: Data sharing in neuroimaging\". <i>Nature Neuroscience<\/i> <b>17<\/b> (11): 1510\u20137. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnn.3818\" target=\"_blank\">10.1038\/nn.3818<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25349916\" target=\"_blank\">25349916<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Making+big+data+open%3A+Data+sharing+in+neuroimaging&rft.jtitle=Nature+Neuroscience&rft.aulast=Poldrack%2C+R.A.%3B+Gorgolewski%2C+K.J.&rft.au=Poldrack%2C+R.A.%3B+Gorgolewski%2C+K.J.&rft.date=2014&rft.volume=17&rft.issue=11&rft.pages=1510%E2%80%937&rft_id=info:doi\/10.1038%2Fnn.3818&rft_id=info:pmid\/25349916&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GorgolewskiFront15-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GorgolewskiFront15_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gorgolewski, K.J.; Varoquaux, G.; Rivera, G. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4392315\" target=\"_blank\">\"NeuroVault.org: A web-based repository for collecting and sharing unthresholded statistical maps of the human brain\"<\/a>. <i>Frontiers in Neuroinformatics<\/i> <b>9<\/b>: 8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffninf.2015.00008\" target=\"_blank\">10.3389\/fninf.2015.00008<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4392315\/\" target=\"_blank\">PMC4392315<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25914639\" target=\"_blank\">25914639<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4392315\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4392315<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=NeuroVault.org%3A+A+web-based+repository+for+collecting+and+sharing+unthresholded+statistical+maps+of+the+human+brain&rft.jtitle=Frontiers+in+Neuroinformatics&rft.aulast=Gorgolewski%2C+K.J.%3B+Varoquaux%2C+G.%3B+Rivera%2C+G.+et+al.&rft.au=Gorgolewski%2C+K.J.%3B+Varoquaux%2C+G.%3B+Rivera%2C+G.+et+al.&rft.date=2015&rft.volume=9&rft.pages=8&rft_id=info:doi\/10.3389%2Ffninf.2015.00008&rft_id=info:pmc\/PMC4392315&rft_id=info:pmid\/25914639&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4392315&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Scheufele_tranSMART14-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Scheufele_tranSMART14_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scheufele, E.; Aronzon, D.; Coopersmith, R. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4333702\" target=\"_blank\">\"tranSMART: An Open Source Knowledge Management and High Content Data Analytics Platform\"<\/a>. <i>AMIA Joint Summits on Translational Science<\/i> <b>2014<\/b>: 96\u2013101. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4333702\/\" target=\"_blank\">PMC4333702<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25717408\" target=\"_blank\">25717408<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4333702\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4333702<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=tranSMART%3A+An+Open+Source+Knowledge+Management+and+High+Content+Data+Analytics+Platform&rft.jtitle=AMIA+Joint+Summits+on+Translational+Science&rft.aulast=Scheufele%2C+E.%3B+Aronzon%2C+D.%3B+Coopersmith%2C+R.+et+al.&rft.au=Scheufele%2C+E.%3B+Aronzon%2C+D.%3B+Coopersmith%2C+R.+et+al.&rft.date=2014&rft.volume=2014&rft.pages=96%E2%80%93101&rft_id=info:pmc\/PMC4333702&rft_id=info:pmid\/25717408&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4333702&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GorgolewskiTheBrain16-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GorgolewskiTheBrain16_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gorgolewski, K.J.; Auer, T.; Calhoun, V.D. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4978148\" target=\"_blank\">\"The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments\"<\/a>. <i>Scientific Data<\/i> <b>3<\/b>: 160044. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fsdata.2016.44\" target=\"_blank\">10.1038\/sdata.2016.44<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4978148\/\" target=\"_blank\">PMC4978148<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27326542\" target=\"_blank\">27326542<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4978148\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4978148<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+brain+imaging+data+structure%2C+a+format+for+organizing+and+describing+outputs+of+neuroimaging+experiments&rft.jtitle=Scientific+Data&rft.aulast=Gorgolewski%2C+K.J.%3B+Auer%2C+T.%3B+Calhoun%2C+V.D.+et+al.&rft.au=Gorgolewski%2C+K.J.%3B+Auer%2C+T.%3B+Calhoun%2C+V.D.+et+al.&rft.date=2016&rft.volume=3&rft.pages=160044&rft_id=info:doi\/10.1038%2Fsdata.2016.44&rft_id=info:pmc\/PMC4978148&rft_id=info:pmid\/27326542&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4978148&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NicholsNeuro15-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NicholsNeuro15_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nichols, B.N.; Pohl, K.M. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5400666\" target=\"_blank\">\"Neuroinformatics Software Applications Supporting Electronic Data Capture, Management, and Sharing for the Neuroimaging Community\"<\/a>. <i>Neuropsychology Review<\/i> <b>25<\/b> (3): 356-68. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11065-015-9293-x\" target=\"_blank\">10.1007\/s11065-015-9293-x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5400666\/\" target=\"_blank\">PMC5400666<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26267019\" target=\"_blank\">26267019<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5400666\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5400666<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Neuroinformatics+Software+Applications+Supporting+Electronic+Data+Capture%2C+Management%2C+and+Sharing+for+the+Neuroimaging+Community&rft.jtitle=Neuropsychology+Review&rft.aulast=Nichols%2C+B.N.%3B+Pohl%2C+K.M.&rft.au=Nichols%2C+B.N.%3B+Pohl%2C+K.M.&rft.date=2015&rft.volume=25&rft.issue=3&rft.pages=356-68&rft_id=info:doi\/10.1007%2Fs11065-015-9293-x&rft_id=info:pmc\/PMC5400666&rft_id=info:pmid\/26267019&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5400666&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HornIsIt09-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HornIsIt09_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Van Horn, J.D.; Toga, A.W. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2754579\" target=\"_blank\">\"Is it time to re-prioritize neuroimaging databases and digital repositories?\"<\/a>. <i>NeuroImage<\/i> <b>47<\/b> (4): 1720-34. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.neuroimage.2009.03.086\" target=\"_blank\">10.1016\/j.neuroimage.2009.03.086<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2754579\/\" target=\"_blank\">PMC2754579<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19371790\" target=\"_blank\">19371790<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2754579\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2754579<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Is+it+time+to+re-prioritize+neuroimaging+databases+and+digital+repositories%3F&rft.jtitle=NeuroImage&rft.aulast=Van+Horn%2C+J.D.%3B+Toga%2C+A.W.&rft.au=Van+Horn%2C+J.D.%3B+Toga%2C+A.W.&rft.date=2009&rft.volume=47&rft.issue=4&rft.pages=1720-34&rft_id=info:doi\/10.1016%2Fj.neuroimage.2009.03.086&rft_id=info:pmc\/PMC2754579&rft_id=info:pmid\/19371790&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2754579&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarcusHuman13-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarcusHuman13_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Marcus, D.S.; Harms, M.P.; Snyder, A.Z. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3845379\" target=\"_blank\">\"Human Connectome Project informatics: quality control, database services, and data visualization\"<\/a>. <i>NeuroImage<\/i> <b>80<\/b>: 202-19. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.neuroimage.2013.05.077\" target=\"_blank\">10.1016\/j.neuroimage.2013.05.077<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3845379\/\" target=\"_blank\">PMC3845379<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23707591\" target=\"_blank\">23707591<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3845379\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3845379<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Human+Connectome+Project+informatics%3A+quality+control%2C+database+services%2C+and+data+visualization&rft.jtitle=NeuroImage&rft.aulast=Marcus%2C+D.S.%3B+Harms%2C+M.P.%3B+Snyder%2C+A.Z.+et+al.&rft.au=Marcus%2C+D.S.%3B+Harms%2C+M.P.%3B+Snyder%2C+A.Z.+et+al.&rft.date=2013&rft.volume=80&rft.pages=202-19&rft_id=info:doi\/10.1016%2Fj.neuroimage.2013.05.077&rft_id=info:pmc\/PMC3845379&rft_id=info:pmid\/23707591&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3845379&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DasLORIS12-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DasLORIS12_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Das, S.; Zijdenbos, A.P.; Harlap, J. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3262165\" target=\"_blank\">\"LORIS: A web-based data management system for multi-center studies\"<\/a>. <i>Frontiers in Neuroinformatics<\/i> <b>5<\/b>: 37. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffninf.2011.00037\" target=\"_blank\">10.3389\/fninf.2011.00037<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3262165\/\" target=\"_blank\">PMC3262165<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22319489\" target=\"_blank\">22319489<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3262165\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3262165<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LORIS%3A+A+web-based+data+management+system+for+multi-center+studies&rft.jtitle=Frontiers+in+Neuroinformatics&rft.aulast=Das%2C+S.%3B+Zijdenbos%2C+A.P.%3B+Harlap%2C+J.+et+al.&rft.au=Das%2C+S.%3B+Zijdenbos%2C+A.P.%3B+Harlap%2C+J.+et+al.&rft.date=2012&rft.volume=5&rft.pages=37&rft_id=info:doi\/10.3389%2Ffninf.2011.00037&rft_id=info:pmc\/PMC3262165&rft_id=info:pmid\/22319489&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3262165&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BookNeuroinfo13-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BookNeuroinfo13_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Book, G.A.; Anderson, B.M.; Stevens, M.C. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3864015\" target=\"_blank\">\"Neuroinformatics Database (NiDB) - A modular, portable database for the storage, analysis, and sharing of neuroimaging data\"<\/a>. <i>Neuroinformatics<\/i> <b>11<\/b> (4): 495-505. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs12021-013-9194-1\" target=\"_blank\">10.1007\/s12021-013-9194-1<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3864015\/\" target=\"_blank\">PMC3864015<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23912507\" target=\"_blank\">23912507<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3864015\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3864015<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Neuroinformatics+Database+%28NiDB%29+-+A+modular%2C+portable+database+for+the+storage%2C+analysis%2C+and+sharing+of+neuroimaging+data&rft.jtitle=Neuroinformatics&rft.aulast=Book%2C+G.A.%3B+Anderson%2C+B.M.%3B+Stevens%2C+M.C.+et+al.&rft.au=Book%2C+G.A.%3B+Anderson%2C+B.M.%3B+Stevens%2C+M.C.+et+al.&rft.date=2013&rft.volume=11&rft.issue=4&rft.pages=495-505&rft_id=info:doi\/10.1007%2Fs12021-013-9194-1&rft_id=info:pmc\/PMC3864015&rft_id=info:pmid\/23912507&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3864015&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OpenClinicaUser16-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OpenClinicaUser16_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/docs.openclinica.com\/\" target=\"_blank\">\"OpenClinica User Documentation\"<\/a>. OpenClinica, LLC. 18 April 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/docs.openclinica.com\/\" target=\"_blank\">https:\/\/docs.openclinica.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OpenClinica+User+Documentation&rft.atitle=&rft.date=18+April+2016&rft.pub=OpenClinica%2C+LLC&rft_id=https%3A%2F%2Fdocs.openclinica.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarrisResearch09-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HarrisResearch09_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Harris, P.A.; Taylor, R.; Thielke, R. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2700030\" target=\"_blank\">\"Research electronic data capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support\"<\/a>. <i>Journal of Biomedical Informatics<\/i> <b>42<\/b> (2): 377\u201381. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2008.08.010\" target=\"_blank\">10.1016\/j.jbi.2008.08.010<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2700030\/\" target=\"_blank\">PMC2700030<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18929686\" target=\"_blank\">18929686<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2700030\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2700030<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Research+electronic+data+capture+%28REDCap%29+-+A+metadata-driven+methodology+and+workflow+process+for+providing+translational+research+informatics+support&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Harris%2C+P.A.%3B+Taylor%2C+R.%3B+Thielke%2C+R.+et+al.&rft.au=Harris%2C+P.A.%3B+Taylor%2C+R.%3B+Thielke%2C+R.+et+al.&rft.date=2009&rft.volume=42&rft.issue=2&rft.pages=377%E2%80%9381&rft_id=info:doi\/10.1016%2Fj.jbi.2008.08.010&rft_id=info:pmc\/PMC2700030&rft_id=info:pmid\/18929686&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2700030&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrestyaninovaASys09-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrestyaninovaASys09_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krestyaninova, M.; Zarins, A.; Viksna, J. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2759553\" target=\"_blank\">\"A system for information management in biomedical studies \u2013 SIMBioMS\"<\/a>. <i>Bioinformatics<\/i> <b>25<\/b> (20): 2768-2769. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp420\" target=\"_blank\">10.1093\/bioinformatics\/btp420<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2759553\/\" target=\"_blank\">PMC2759553<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19633095\" target=\"_blank\">19633095<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2759553\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2759553<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+system+for+information+management+in+biomedical+studies+%E2%80%93+SIMBioMS&rft.jtitle=Bioinformatics&rft.aulast=Krestyaninova%2C+M.%3B+Zarins%2C+A.%3B+Viksna%2C+J.+et+al.&rft.au=Krestyaninova%2C+M.%3B+Zarins%2C+A.%3B+Viksna%2C+J.+et+al.&rft.date=2009&rft.volume=25&rft.issue=20&rft.pages=2768-2769&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp420&rft_id=info:pmc\/PMC2759553&rft_id=info:pmid\/19633095&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2759553&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScottCOINS11-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScottCOINS11_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scott, A.; Courtney, W.; Wood, D. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3250631\" target=\"_blank\">\"COINS: An Innovative Informatics and Neuroimaging Tool Suite Built for Large Heterogeneous Datasets\"<\/a>. <i>Frontiers in Neuroinformatics<\/i> <b>5<\/b>: 33. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffninf.2011.00033\" target=\"_blank\">10.3389\/fninf.2011.00033<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3250631\/\" target=\"_blank\">PMC3250631<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22275896\" target=\"_blank\">22275896<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3250631\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3250631<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=COINS%3A+An+Innovative+Informatics+and+Neuroimaging+Tool+Suite+Built+for+Large+Heterogeneous+Datasets&rft.jtitle=Frontiers+in+Neuroinformatics&rft.aulast=Scott%2C+A.%3B+Courtney%2C+W.%3B+Wood%2C+D.+et+al.&rft.au=Scott%2C+A.%3B+Courtney%2C+W.%3B+Wood%2C+D.+et+al.&rft.date=2011&rft.volume=5&rft.pages=33&rft_id=info:doi\/10.3389%2Ffninf.2011.00033&rft_id=info:pmc\/PMC3250631&rft_id=info:pmid\/22275896&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3250631&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LogilabCubicWeb-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LogilabCubicWeb_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-LogilabCubicWeb_15-1\" rel=\"external_link\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-LogilabCubicWeb_15-2\" rel=\"external_link\">15.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cubicweb.org\/\" target=\"_blank\">\"CubicWeb - The Semantic Web is a construction game!\"<\/a>. Logilab. 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cubicweb.org\/\" target=\"_blank\">https:\/\/www.cubicweb.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CubicWeb+-+The+Semantic+Web+is+a+construction+game%21&rft.atitle=&rft.date=2016&rft.pub=Logilab&rft_id=https%3A%2F%2Fwww.cubicweb.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchumannTheIMAGEN10-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SchumannTheIMAGEN10_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-SchumannTheIMAGEN10_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schumann, G.; Loth, E.; Banaschewski, T. et al. (2010). \"The IMAGEN study: Reinforcement-related behaviour in normal brain function and psychopathology\". <i>Molecular Psychiatry<\/i> <b>15<\/b> (12): 1128-39. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fmp.2010.4\" target=\"_blank\">10.1038\/mp.2010.4<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21102431\" target=\"_blank\">21102431<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+IMAGEN+study%3A+Reinforcement-related+behaviour+in+normal+brain+function+and+psychopathology&rft.jtitle=Molecular+Psychiatry&rft.aulast=Schumann%2C+G.%3B+Loth%2C+E.%3B+Banaschewski%2C+T.+et+al.&rft.au=Schumann%2C+G.%3B+Loth%2C+E.%3B+Banaschewski%2C+T.+et+al.&rft.date=2010&rft.volume=15&rft.issue=12&rft.pages=1128-39&rft_id=info:doi\/10.1038%2Fmp.2010.4&rft_id=info:pmid\/21102431&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MurphyEU-AIMS12-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MurphyEU-AIMS12_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-MurphyEU-AIMS12_17-1\" rel=\"external_link\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Murphy, D.; Spooren, W. (2012). \"EU-AIMS: A boost to autism research\". <i>Nature Reviews Drug Discovery<\/i> <b>11<\/b> (11): 815-6. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnrd3881\" target=\"_blank\">10.1038\/nrd3881<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23123927\" target=\"_blank\">23123927<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EU-AIMS%3A+A+boost+to+autism+research&rft.jtitle=Nature+Reviews+Drug+Discovery&rft.aulast=Murphy%2C+D.%3B+Spooren%2C+W.&rft.au=Murphy%2C+D.%3B+Spooren%2C+W.&rft.date=2012&rft.volume=11&rft.issue=11&rft.pages=815-6&rft_id=info:doi\/10.1038%2Fnrd3881&rft_id=info:pmid\/23123927&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MichelBrainomics13-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MichelBrainomics13_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Michel, V.; Schwartz, Y.; Pinel, P. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/hal.inria.fr\/cea-00904768\/en\" target=\"_blank\">\"Brainomics: A management system for exploring and merging heterogeneous brain mapping data\"<\/a>. <i>Proceedings from the 19th Annual Meeting of the Organization for Human Brain Mapping<\/i> <b>2013<\/b><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/hal.inria.fr\/cea-00904768\/en\" target=\"_blank\">https:\/\/hal.inria.fr\/cea-00904768\/en<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Brainomics%3A+A+management+system+for+exploring+and+merging+heterogeneous+brain+mapping+data&rft.jtitle=Proceedings+from+the+19th+Annual+Meeting+of+the+Organization+for+Human+Brain+Mapping&rft.aulast=Michel%2C+V.%3B+Schwartz%2C+Y.%3B+Pinel%2C+P.+et+al.&rft.au=Michel%2C+V.%3B+Schwartz%2C+Y.%3B+Pinel%2C+P.+et+al.&rft.date=2013&rft.volume=2013&rft_id=https%3A%2F%2Fhal.inria.fr%2Fcea-00904768%2Fen&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PapadopoulosTheBrainomics17-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PapadopoulosTheBrainomics17_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Papadopoulos Orfanos, D.; Michel, V.; Schwartz, Y. et al. (2017). \"The Brainomics\/Localizer database\". <i>NeuroImage<\/i> <b>144<\/b> (Pt B): 309-314. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.neuroimage.2015.09.052\" target=\"_blank\">10.1016\/j.neuroimage.2015.09.052<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26455807\" target=\"_blank\">26455807<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Brainomics%2FLocalizer+database&rft.jtitle=NeuroImage&rft.aulast=Papadopoulos+Orfanos%2C+D.%3B+Michel%2C+V.%3B+Schwartz%2C+Y.+et+al.&rft.au=Papadopoulos+Orfanos%2C+D.%3B+Michel%2C+V.%3B+Schwartz%2C+Y.+et+al.&rft.date=2017&rft.volume=144&rft.issue=Pt+B&rft.pages=309-314&rft_id=info:doi\/10.1016%2Fj.neuroimage.2015.09.052&rft_id=info:pmid\/26455807&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Prud.27hommeauxSPARQL08-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Prud.27hommeauxSPARQL08_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Prud'hommeaux, E.; Seaborne, A., ed. (15 January 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.w3.org\/TR\/rdf-sparql-query\/\" target=\"_blank\">\"SPARQL Query Language for RDF\"<\/a>. World Wide Web Consortium<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.w3.org\/TR\/rdf-sparql-query\/\" target=\"_blank\">https:\/\/www.w3.org\/TR\/rdf-sparql-query\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SPARQL+Query+Language+for+RDF&rft.atitle=&rft.date=15+January+2008&rft.pub=World+Wide+Web+Consortium&rft_id=https%3A%2F%2Fwww.w3.org%2FTR%2Frdf-sparql-query%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GorgolewskiNipype11-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GorgolewskiNipype11_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gorgolewski, K.; Burns, C.D.; Madison, C. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3159964\" target=\"_blank\">\"Nipype: A flexible, lightweight and extensible neuroimaging data processing framework in Python\"<\/a>. <i>Frontiers in Neuroinformatics<\/i> <b>5<\/b> (Pt B): 13. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffninf.2011.00013\" target=\"_blank\">10.3389\/fninf.2011.00013<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3159964\/\" target=\"_blank\">PMC3159964<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21897815\" target=\"_blank\">21897815<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3159964\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3159964<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Nipype%3A+A+flexible%2C+lightweight+and+extensible+neuroimaging+data+processing+framework+in+Python&rft.jtitle=Frontiers+in+Neuroinformatics&rft.aulast=Gorgolewski%2C+K.%3B+Burns%2C+C.D.%3B+Madison%2C+C.+et+al.&rft.au=Gorgolewski%2C+K.%3B+Burns%2C+C.D.%3B+Madison%2C+C.+et+al.&rft.date=2011&rft.volume=5&rft.issue=Pt+B&rft.pages=13&rft_id=info:doi\/10.3389%2Ffninf.2011.00013&rft_id=info:pmc\/PMC3159964&rft_id=info:pmid\/21897815&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3159964&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChapmanBiopython00-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChapmanBiopython00_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chapman, B.; Chang, J. (2000). \"Biopython: Python tools for computational biology\". <i>ACM SIGBIO Newsletter<\/i> <b>20<\/b> (2): 15\u201319. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F360262.360268\" target=\"_blank\">10.1145\/360262.360268<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Biopython%3A+Python+tools+for+computational+biology&rft.jtitle=ACM+SIGBIO+Newsletter&rft.aulast=Chapman%2C+B.%3B+Chang%2C+J.&rft.au=Chapman%2C+B.%3B+Chang%2C+J.&rft.date=2000&rft.volume=20&rft.issue=2&rft.pages=15%E2%80%9319&rft_id=info:doi\/10.1145%2F360262.360268&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbrahamMachine14-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbrahamMachine14_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abraham, A.; Pedregosa, F.; Eickenberg, M. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3930868\" target=\"_blank\">\"Machine learning for neuroimaging with Scikit-learn\"<\/a>. <i>Frontiers in Neuroinformatics<\/i> <b>8<\/b>: 14. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffninf.2014.00014\" target=\"_blank\">10.3389\/fninf.2014.00014<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3930868\/\" target=\"_blank\">PMC3930868<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24600388\" target=\"_blank\">24600388<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3930868\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3930868<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Machine+learning+for+neuroimaging+with+Scikit-learn&rft.jtitle=Frontiers+in+Neuroinformatics&rft.aulast=Abraham%2C+A.%3B+Pedregosa%2C+F.%3B+Eickenberg%2C+M.+et+al.&rft.au=Abraham%2C+A.%3B+Pedregosa%2C+F.%3B+Eickenberg%2C+M.+et+al.&rft.date=2014&rft.volume=8&rft.pages=14&rft_id=info:doi\/10.3389%2Ffninf.2014.00014&rft_id=info:pmc\/PMC3930868&rft_id=info:pmid\/24600388&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3930868&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FischerMorph12-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FischerMorph12_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fischer, C.; Operto, G.; Laguitton, S. et al.. \"Morphologist 2012: The new morphological pipeline of BrainVisa\". <i>Proceedings from the Human Brain Mapping Conference 2012<\/i> <b>2012<\/b>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Morphologist+2012%3A+The+new+morphological+pipeline+of+BrainVisa&rft.jtitle=Proceedings+from+the+Human+Brain+Mapping+Conference+2012&rft.aulast=Fischer%2C+C.%3B+Operto%2C+G.%3B+Laguitton%2C+S.+et+al.&rft.au=Fischer%2C+C.%3B+Operto%2C+G.%3B+Laguitton%2C+S.+et+al.&rft.volume=2012&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubFUSE-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GitHubFUSE_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/libfuse\/libfuse\" target=\"_blank\">\"libfuse\/libfuse\"<\/a>. GitHub, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/libfuse\/libfuse\" target=\"_blank\">https:\/\/github.com\/libfuse\/libfuse<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=libfuse%2Flibfuse&rft.atitle=&rft.pub=GitHub%2C+Inc&rft_id=https%3A%2F%2Fgithub.com%2Flibfuse%2Flibfuse&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RockholdExtract12-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RockholdExtract12_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rockhold, F.; Bishop, S. (2012). \"Extracting the value of standards: The role of CDISC in a pharmaceutical research strategy\". <i>Clinical Evaluation<\/i> <b>40<\/b>: 91\u201396.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Extracting+the+value+of+standards%3A+The+role+of+CDISC+in+a+pharmaceutical+research+strategy&rft.jtitle=Clinical+Evaluation&rft.aulast=Rockhold%2C+F.%3B+Bishop%2C+S.&rft.au=Rockhold%2C+F.%3B+Bishop%2C+S.&rft.date=2012&rft.volume=40&rft.pages=91%E2%80%9396&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NSApRQLUpload-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NSApRQLUpload_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">NSAp developers. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/neurospin.github.io\/rql_upload\/\" target=\"_blank\">\"Rql Upload\"<\/a>. GitHub, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/neurospin.github.io\/rql_upload\/\" target=\"_blank\">http:\/\/neurospin.github.io\/rql_upload\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Rql+Upload&rft.atitle=&rft.aulast=NSAp+developers&rft.au=NSAp+developers&rft.pub=GitHub%2C+Inc&rft_id=http%3A%2F%2Fneurospin.github.io%2Frql_upload%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NSApPIWS-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NSApPIWS_28-0\" rel=\"external_link\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-NSApPIWS_28-1\" rel=\"external_link\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">NSAp developers. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/neurospin.github.io\/piws\/\" target=\"_blank\">\"Population Imaging Web Service: PIWS\"<\/a>. GitHub, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/neurospin.github.io\/piws\/\" target=\"_blank\">http:\/\/neurospin.github.io\/piws\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Population+Imaging+Web+Service%3A+PIWS&rft.atitle=&rft.aulast=NSAp+developers&rft.au=NSAp+developers&rft.pub=GitHub%2C+Inc&rft_id=http%3A%2F%2Fneurospin.github.io%2Fpiws%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubZeijemol-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GitHubZeijemol_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/neurospin\/zeijemol\" target=\"_blank\">\"neurospin\/zeijemol\"<\/a>. GitHub, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/neurospin\/zeijemol\" target=\"_blank\">https:\/\/github.com\/neurospin\/zeijemol<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=neurospin%2Fzeijemol&rft.atitle=&rft.pub=GitHub%2C+Inc&rft_id=https%3A%2F%2Fgithub.com%2Fneurospin%2Fzeijemol&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NSApRQLDownload-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NSApRQLDownload_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">NSAp developers. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/neurospin.github.io\/rql_download\/\" target=\"_blank\">\"Rql Download\"<\/a>. GitHub, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/neurospin.github.io\/rql_download\/\" target=\"_blank\">http:\/\/neurospin.github.io\/rql_download\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Rql+Download&rft.atitle=&rft.aulast=NSAp+developers&rft.au=NSAp+developers&rft.pub=GitHub%2C+Inc&rft_id=http%3A%2F%2Fneurospin.github.io%2Frql_download%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GitHubBrainomics2-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GitHubBrainomics2_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/neurospin\/brainomics2\" target=\"_blank\">\"neurospin\/brainomics2\"<\/a>. GitHub, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/neurospin\/brainomics2\" target=\"_blank\">https:\/\/github.com\/neurospin\/brainomics2<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=neurospin%2Fbrainomics2&rft.atitle=&rft.pub=GitHub%2C+Inc&rft_id=https%3A%2F%2Fgithub.com%2Fneurospin%2Fbrainomics2&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SFCTwisted-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SFCTwisted_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/twistedmatrix.com\/trac\/\" target=\"_blank\">\"What is Twisted?\"<\/a>. Software Freedom Conservancy<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/twistedmatrix.com\/trac\/\" target=\"_blank\">https:\/\/twistedmatrix.com\/trac\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+is+Twisted%3F&rft.atitle=&rft.pub=Software+Freedom+Conservancy&rft_id=https%3A%2F%2Ftwistedmatrix.com%2Ftrac%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchwartzPyXNAT12-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchwartzPyXNAT12_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schwartz, Y.; Barbot, A.; Thyreau, B. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3354345\" target=\"_blank\">\"PyXNAT: XNAT in Python\"<\/a>. <i>Frontiers in Neuroinformatics<\/i> <b>6<\/b>: 12. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffninf.2012.00012\" target=\"_blank\">10.3389\/fninf.2012.00012<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3354345\/\" target=\"_blank\">PMC3354345<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22654752\" target=\"_blank\">22654752<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3354345\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3354345<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PyXNAT%3A+XNAT+in+Python&rft.jtitle=Frontiers+in+Neuroinformatics&rft.aulast=Schwartz%2C+Y.%3B+Barbot%2C+A.%3B+Thyreau%2C+B.+et+al.&rft.au=Schwartz%2C+Y.%3B+Barbot%2C+A.%3B+Thyreau%2C+B.+et+al.&rft.date=2012&rft.volume=6&rft.pages=12&rft_id=info:doi\/10.3389%2Ffninf.2012.00012&rft_id=info:pmc\/PMC3354345&rft_id=info:pmid\/22654752&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3354345&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NSApCWBROWSER-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NSApCWBROWSER_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">NSAp developers. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/neurospin.github.io\/rql_download\/cwbrowser\" target=\"_blank\">\"CWBROWSER\"<\/a>. GitHub, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/neurospin.github.io\/rql_download\/cwbrowser\" target=\"_blank\">http:\/\/neurospin.github.io\/rql_download\/cwbrowser<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CWBROWSER&rft.atitle=&rft.aulast=NSAp+developers&rft.au=NSAp+developers&rft.pub=GitHub%2C+Inc&rft_id=http%3A%2F%2Fneurospin.github.io%2Frql_download%2Fcwbrowser&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KeatorTowards13-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KeatorTowards13_35-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Keator, D.B.; Helmer, K.; Steffener, J. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4028152\" target=\"_blank\">\"Towards structured sharing of raw and derived neuroimaging data across existing resources\"<\/a>. <i>Neuroimage<\/i> <b>82<\/b>: 647-61. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.neuroimage.2013.05.094\" target=\"_blank\">10.1016\/j.neuroimage.2013.05.094<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4028152\/\" target=\"_blank\">PMC4028152<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23727024\" target=\"_blank\">23727024<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4028152\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4028152<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+structured+sharing+of+raw+and+derived+neuroimaging+data+across+existing+resources&rft.jtitle=Neuroimage&rft.aulast=Keator%2C+D.B.%3B+Helmer%2C+K.%3B+Steffener%2C+J.+et+al.&rft.au=Keator%2C+D.B.%3B+Helmer%2C+K.%3B+Steffener%2C+J.+et+al.&rft.date=2013&rft.volume=82&rft.pages=647-61&rft_id=info:doi\/10.1016%2Fj.neuroimage.2013.05.094&rft_id=info:pmc\/PMC4028152&rft_id=info:pmid\/23727024&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4028152&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PoldrackTheCog11-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PoldrackTheCog11_36-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Poldrack, R.A.; Kittur, A.; Kalar, D. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3167196\" target=\"_blank\">\"The cognitive atlas: Toward a knowledge foundation for cognitive neuroscience\"<\/a>. <i>Frotiers in Neuroinformatics<\/i> <b>5<\/b>: 17. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffninf.2011.00017\" target=\"_blank\">10.3389\/fninf.2011.00017<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3167196\/\" target=\"_blank\">PMC3167196<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21922006\" target=\"_blank\">21922006<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3167196\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3167196<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+cognitive+atlas%3A+Toward+a+knowledge+foundation+for+cognitive+neuroscience&rft.jtitle=Frotiers+in+Neuroinformatics&rft.aulast=Poldrack%2C+R.A.%3B+Kittur%2C+A.%3B+Kalar%2C+D.+et+al.&rft.au=Poldrack%2C+R.A.%3B+Kittur%2C+A.%3B+Kalar%2C+D.+et+al.&rft.date=2011&rft.volume=5&rft.pages=17&rft_id=info:doi\/10.3389%2Ffninf.2011.00017&rft_id=info:pmc\/PMC3167196&rft_id=info:pmid\/21922006&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3167196&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GibaudNeuroLOG11-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GibaudNeuroLOG11_37-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gibaud, B.; Kassel, G.; Dojat, M. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3243145\" target=\"_blank\">\"NeuroLOG: Sharing neuroimaging data using an ontology-based federated approach\"<\/a>. <i>AMIA Annual Symposium Proceedings<\/i> <b>2011<\/b>: 472-80. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3243145\/\" target=\"_blank\">PMC3243145<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22195101\" target=\"_blank\">22195101<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3243145\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3243145<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=NeuroLOG%3A+Sharing+neuroimaging+data+using+an+ontology-based+federated+approach&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Gibaud%2C+B.%3B+Kassel%2C+G.%3B+Dojat%2C+M.+et+al.&rft.au=Gibaud%2C+B.%3B+Kassel%2C+G.%3B+Dojat%2C+M.+et+al.&rft.date=2011&rft.volume=2011&rft.pages=472-80&rft_id=info:pmc\/PMC3243145&rft_id=info:pmid\/22195101&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3243145&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DumontierBio2RDF14-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DumontierBio2RDF14_38-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dumontier, M.; Callahan, A.; Cruz-Toledo, J. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ceur-ws.org\/Vol-1272\/paper_121.pdf\" target=\"_blank\">\"Bio2RDF release 3: A larger connected network of linked data for the life sciences\"<\/a>. <i>Proceedings of the 2014 International Conference on Posters & Demonstrations Track<\/i> <b>1272<\/b>: 401\u201304<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ceur-ws.org\/Vol-1272\/paper_121.pdf\" target=\"_blank\">http:\/\/ceur-ws.org\/Vol-1272\/paper_121.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bio2RDF+release+3%3A+A+larger+connected+network+of+linked+data+for+the+life+sciences&rft.jtitle=Proceedings+of+the+2014+International+Conference+on+Posters+%26+Demonstrations+Track&rft.aulast=Dumontier%2C+M.%3B+Callahan%2C+A.%3B+Cruz-Toledo%2C+J.+et+al.&rft.au=Dumontier%2C+M.%3B+Callahan%2C+A.%3B+Cruz-Toledo%2C+J.+et+al.&rft.date=2014&rft.volume=1272&rft.pages=401%E2%80%9304&rft_id=http%3A%2F%2Fceur-ws.org%2FVol-1272%2Fpaper_121.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. References are in order of appearance rather than alphabetical order (as the original was) due to the way this wiki works. Footnotes (URLs to projects) were turned into full citations. The URL to the <i>zeijemol<\/i> application cube was broken, and a direct GitHub URL was substituted instead.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181500\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.841 seconds\nReal time usage: 0.869 seconds\nPreprocessor visited node count: 29459\/1000000\nPreprocessor generated node count: 35265\/1000000\nPost\u2010expand include size: 239646\/2097152 bytes\nTemplate argument size: 77434\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 824.291 1 - -total\n 87.09% 717.883 1 - Template:Reflist\n 77.31% 637.261 38 - Template:Citation\/core\n 63.20% 520.929 27 - Template:Cite_journal\n 17.92% 147.736 11 - Template:Cite_web\n 8.97% 73.930 61 - Template:Citation\/identifier\n 8.23% 67.824 1 - Template:Infobox_journal_article\n 7.95% 65.567 1 - Template:Infobox\n 4.85% 39.978 80 - Template:Infobox\/row\n 4.47% 36.819 38 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10148-0!*!0!!en!5!* and timestamp 20181214181459 and revision id 30787\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework\">https:\/\/www.limswiki.org\/index.php\/Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","bce85c098ea6958c92b6dcce94e42565_images":["https:\/\/www.limswiki.org\/images\/6\/63\/Fig1_Grigis_FInNeuroinformatics2017_11.jpg","https:\/\/www.limswiki.org\/images\/3\/33\/Fig2_Grigis_FInNeuroinformatics2017_11.jpg","https:\/\/www.limswiki.org\/images\/6\/6a\/Fig3_Grigis_FInNeuroinformatics2017_11.jpg","https:\/\/www.limswiki.org\/images\/f\/fa\/Fig4_Grigis_FInNeuroinformatics2017_11.jpg","https:\/\/www.limswiki.org\/images\/1\/1b\/Fig5_Grigis_FInNeuroinformatics2017_11.jpg","https:\/\/www.limswiki.org\/images\/c\/c9\/Fig6_Grigis_FInNeuroinformatics2017_11.jpg"],"bce85c098ea6958c92b6dcce94e42565_timestamp":1544811299,"e86f9fcd9da316970f965a883c42e462_type":"article","e86f9fcd9da316970f965a883c42e462_title":"A multi-service data management platform for scientific oceanographic products (D\u2019Anca et al. 2017)","e86f9fcd9da316970f965a883c42e462_url":"https:\/\/www.limswiki.org\/index.php\/Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products","e86f9fcd9da316970f965a883c42e462_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:A multi-service data management platform for scientific oceanographic products\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nA multi-service data management platform for scientific oceanographic productsJournal\n \nNatural Hazards and Earth System SciencesAuthor(s)\n \nD'Anca, Alessandro; Conte, Laura; Nassisi, Paola; Palazzo, Cosimo; Lecci, Rita; Cret\u00ec, Sergio; Mancini, Marco;\r\nNuzzo, Alessandra; Mirto, Maria; Mannarini, Gianandrea; Coppini, Giovanni; Fiore, Sandro; Aloisio, GiovanniAuthor affiliation(s)\n \nCentro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC), University of SalentoPrimary contact\n \nEmail: alessandro dot danca at cmcc dot itEditors\n \nMarra, P.Year published\n \n2017Volume and issue\n \n17(2)Page(s)\n \n171-184DOI\n \n10.5194\/nhess-17-171-2017ISSN\n \n1684-9981Distribution license\n \nCreative Commons Attribution 3.0Website\n \nhttp:\/\/www.nat-hazards-earth-syst-sci.net\/17\/171\/2017\/nhess-17-171-2017.htmlDownload\n \nhttp:\/\/www.nat-hazards-earth-syst-sci.net\/17\/171\/2017\/nhess-17-171-2017.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Related work \n4 The data platform architecture \n5 The Data Access Service \n\n5.1 The dataset repository: The TESSA data archive \n5.2 Implementation of the DAS-DTS modules \n\n\n6 The Metadata Service and the metadata profile \n\n6.1 Implementation of the Metadata Service \n\n\n7 The Complex Data Analysis Module \n\n7.1 The CDAM gateway \n7.2 The CDAM launcher \n\n\n8 Operational activity: TESSA data platform use cases \n\n8.1 Automatic publishing procedure on the ESGF data node \n8.2 DTS automatic procedures for DSS and numerical models transformations \n8.3 SeaConditions operational chain \n\n\n9 Conclusions \n10 Data availability \n11 Acknowledgements \n12 Competing interests \n13 References \n14 Notes \n\n\n\nAbstract \nAn efficient, secure and interoperable data platform solution has been developed in the TESSA project to provide fast navigation and access to the data stored in the data archive, as well as a standard-based metadata management support. The platform mainly targets scientific users and the situational sea awareness high-level services such as the decision support systems (DSS). These datasets are accessible through the following three main components: the Data Access Service (DAS), the Metadata Service and the Complex Data Analysis Module (CDAM). The DAS allows access to data stored in the archive by providing interfaces for different protocols and services for downloading, variable selection, data subsetting or map generation. Metadata Service is the heart of the information system of TESSA products and completes the overall infrastructure for data and metadata management. This component enables data search and discovery and addresses interoperability by exploiting widely adopted standards for geospatial data. Finally, the CDAM represents the back end of the TESSA DSS by performing on-demand complex data analysis tasks.\n\nIntroduction \nTESSA (Development of TEchnology for Situational Sea Awareness) is a research project born from the collaboration between operational oceanography research and scientific computing groups in order to strengthen operational oceanography capabilities in Southern Italy for use by end users in the maritime, tourism and environmental protection sectors. This project has been very innovative as it has provided the integration of marine and ocean forecasts and analyses with advanced technological platforms. Specifically, an efficient, secure and interoperable data platform solution has been developed to provide fast navigation and access to the data stored in the data archive, as well as standard-based metadata management support. \nThis platform mainly targets scientific users and contains a set of high-level services such as the decision support systems (DSS) for supporting end users in managing emergency situations due to natural hazards in the Mediterranean Sea. For example, the DSS WITOIL[1][2] is crucial for oil spill accidents which could have severe impacts on the Mediterranean Basin and contribute effectively to the reduction of natural disaster risks. Moreover, the DSS Ocean-SAR[3] supports the search-and-rescue (SAR) operations following accidents, and EarlyWarning manages alerts in cases of extreme events by providing near-real-time information on weather and oceanographic conditions. Finally, VISIR[4][5] is able to compute the optimal ship route with the aim of increasing safety and efficiency of navigation. It relies on various forecast environmental fields which affect the vessel's stability in order to ensure that the computed route does not result in an exposure to dynamical hazards. In this context, the developed platform faces the lack of an efficient dissemination of marine environmental data in order to support situational sea awareness (SSA), which is strategically important for maritime safety and security.[6] In fact, an updated situation awareness requires an advanced technological system to make data available for decision makers, improving the capacity of intervention and avoiding potential damages.\nIn a \"data-centric\" perspective \u2014 in which different services, applications or users make use of the outputs of regional or global numerical models \u2014 the TESSA data platform meets the request of near-real-time access to heterogeneous data with different accuracy, resolution or degrees of aggregation. The design phase has been driven by multiple needs that the developed solution had to satisfy. First of all, nowadays each data management system should satisfy the FAIR guiding principles for scientific data management and stewardship[7]: they face the lack of a shared, comprehensive, broadly applicable and articulated guideline concerning the publication of scientific data. Specifically, they identify findability, accessibility, interoperability and reusability as the four principles which need to be addressed by data (and associated metadata) to be considered a good publication. As explained later, the employment of well-known standards concerning protocols, services and output formats satisfies these requirements. In addition, the need for a service that provides information about sea conditions 24\/7 at high and very high spatial and temporal resolution has been addressed by exploiting high-performance and high-availability hardware and software solutions. The developed platform must be able to support the requests of intermediate and common users. To this end, data must be available in the native and standard format (NetCDF)[8][9] as output of the oceanographic models, through a simple and intuitive platform suitable for machine-based interactions, in order to feed user-friendly services for displaying clear maps and graphs. At the end, the system has to provide on-demand services to support decisions; the users must be able to interact with the datasets produced by the models in near-real time. As such, the platform has to provide services and datasets suitable for on-demand processing while minimizing the downloading time and the related input file size.\nThis paper is organized as follows: The next section presents a survey on related work, and in the following section \"The data platform architecture\" an architectural overview of the main data platform components is provided. The focus is on providing easier and unified access to the heterogeneous data produced in the framework of the TESSA project. The implemented data archive and the modules that make the datasets available to the entire set of services are presented in \"The Data Access Service.\" In \"The Metadata Service and the metadata profile,\" the Metadata Service features are detailed from a methodological and technical point-of-view. The module developed in order to serve remote DSS submission requests is described in \"The Complex Data Analysis Module.\" Finally in the final section on operational activity, a few use cases are presented, highlighting the operational chains which exploit the data platform services.\n\nRelated work \nThe need for marine and oceanic data management supporting the SSA and the operational oceanography has led to the definition and development of various platforms that provide different types of data and services. In this context, the MyOcean project[10][11] can be mentioned, as this project implementation was in line with the best practices of the GMES\/Copernicus framework.[12]\nSpecifically, regarding data access, MyOcean provides the scientific community with a unified interface designed to take into account various international standards (ISO 19115[13], ISO 19139[14] and INSPIRE[15][16]). Concerning data management, MyOcean relies on OPeNDAP\/THREDDS for tasks like map subsetting and FTP for direct download. In addition, a valid solution is represented by the Earth System Grid Federation (ESGF)[17][18], a federated system used as metadata service with advanced features, that will be described later. EMODNET MEDSEA Checkpoint[19][20] is another solution supporting data collection and data search and discovery: it exploits a checkpoint browser and a checkpoint dashboard, which presents indicators automatically produced from information databases. SeaDataNet[21] represents a distributed marine data management infrastructure able to manage different large datasets related to in situ and remote observation of the seas and oceans. Through a distributed network approach, it provides an integrated overview and access to datasets provided by 90 national oceanographic and marine data centers. Finally, another efficient solution has been developed within the project CLIPC.[22] It provides access to climate information, including data from satellite and in situ observations, transformed data products and climate change impact indicators. Moreover, users can exploit the CLIPC toolbox to generate, compare, manipulate and combine indicators, create a user basket or launch new jobs for indices calculation.\nThese systems support a wide range of functionalities such as data access, classification, search and discovery and downloading by means of a distributed architecture. Graphs and maps production are also supported, and some of these propose tools to manipulate and combine datasets. The proposed architecture of the TESSA data platform should offer superior and fast data management functionalities, producing datasets suitable for serving different kinds of services in near-real time; access, search and discovery are just some of the features offered. The developed system, in fact, provides a common interface for submitting complex algorithms in a high-performance computing environment using standard approaches and formats exploiting a unified infrastructure for supporting different scenarios and applications.\n\nThe data platform architecture \nAs seen in Fig. 1, the TESSA data platform is composed of three main components: the Data Access Service (DAS), the Metadata Service and the Complex Data Analysis Module (CDAM).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. The TESSA data platform\n\n\n\nThe DAS allows the access to the data archive, the storage of the data produced in the framework of the TESSA project. The main consumers of these services are the \u201cintermediate users\u201d (the scientific community) and the automatic procedures fed by produced datasets to perform the needed transformations. In this context, the DAS represents the single interface to access the data archive of TESSA; its design has been driven by the aim to satisfy requirements such as transparency, robustness, efficiency, fault tolerance, security, data delivery, subsetting and browsing. This implies that this component must necessarily be \"data-centric\" and \"multi-service\" in order to offer access to the data, often stored in a single instance on the data archive, through different protocols and, therefore, different levels of functionality.\nIn addition, the need of interoperability with other data management platforms has driven the design toward the inclusion of well-known services:\n\n\u2013 \"domain-oriented,\" able to provide specific functionalities for the climate change domain; and\n\u2013 \"general purpose,\" to satisfy general requirements exploiting common services and protocols.\nThe Metadata Service has been designed as the heart of the information system of the TESSA products, to support the activities of search and discovery and metadata management. Moreover, the Metadata Service addresses the requirements of interoperability, considered a key feature in order to provide to any potential user easier and unified access to the different types of data produced by the project. These requirements have been satisfied through the adoption of a metadata profile compliant to the ISO standards for geospatial data[23] and the implementation of an efficient information system relying on a specific ESGF data node\nFinally, the CDAM supports the SSA high-level services by performing on-demand complex analysis on the datasets stored in the data archive. In particular, a number of intelligent decisions support systems for complex dynamical environments have been designed and developed, exploiting high-performance computing functionalities to execute their algorithms. One of the goals is to deploy an infrastructure where a DSS requires the execution of a complex task including (i) accessing data stored in the archive, (ii) executing one or more preliminary tasks for processing them and (iii) storing the results in a specific section of the data archive or publishing them on other sites. Each DSS request interacts with the platform involving a three-layer infrastructure: the user client (web or mobile platform), the SSA platform[24] and the CDAM. Each layer communicates with their linked tiers exchanging messages using well-known protocols, and it has been developed in order to implement a high separation of concerns.\nThe design and the main features of these modules will be detailed in the following sections.\n\nThe Data Access Service \nAs stated in the previous section, the DAS represents the single interface to access the data archive of TESSA; specifically, it has been designed with the aim to set up a multi-service platform to access and analyze scientific data. Among the exposed services, it is important to mention the THREDDS Data Server (TDS)[25], which represents a web server that provides metadata and data access for scientific datasets, using a variety of remote data access protocols. More in detail, it offers a set of services that allows users direct access to data produced in the context of the TESSA project, namely the output of the AFS (Adriatic Forecasting System)[26][27][28], AIFS[29][30] and SANIFS[31] sub-regional models. In particular, the THREDDS includes the WMS (Web Map Service)[32] for the dynamic generation of maps created from geographically referenced data and directly downloaded using the HTTP protocol. Furthermore, it includes the OPeNDAP[33] service, which is a framework that aims to simplify the sharing of scientific information on the web by making available local data from remote connections, providing also variable selection and spatial and temporal subsetting capabilities. To improve performance in terms of load balancing and increase the fault tolerance of the system, a multiplexed configuration for the THREDDS installation based on two different hosts has been used. In addition, a sub-component, the Data Transformation Service (DTS), represents a middleware responsible for applying different data transformations in order to make data compliant to the different protocols offered by the DAS and therefore available to the data consumers (users or services) in a transparent way.\n\nThe dataset repository: The TESSA data archive \nThe data archive represents the storage for all the data produced within the TESSA project. It has been designed and implemented in order to ease the data search-and-discovery phase exploiting their classification while, from an infrastructure point of view, data are stored on a GPFS[34] file system (in RAID 6 configuration), able to provide fast data access and a high level of fault tolerance. Figure 2 shows a high-level schema of the archive.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. High-level representation of the data archive\n\n\n\nThe data can be subdivided into two categories:\n\n\u2013 historical data, which are permanently stored into the archive, including observations and model reanalysis and best analysis; and\n\u2013 rolling data, which are subjected to a rolling procedure, including model worst analysis, simulations and forecasts.\nConsidering the different kinds of data, the data archive has been created following a tree-based topology where the directory names reflect the data classification (historical as first level, model or observation as second level, atmospheric or oceanographic as third level, etc.). The directory tree includes also the institute providing the data, the name of the system that produced the data and the temporal resolution of the data. This kind of classification eases and speeds up the search and discovery of data by intermediate\/scientific users and automatic procedures. Concerning the \"historical\" archive, data stored under \"obs\" include observations retrieved by different instruments (satellites, in situ buoys, vessels, etc.) and consider measurements of the main oceanographic variables (temperature, altimetry, salinity, chlorophyll concentration, etc.). The spatial and the temporal resolution and the temporal coverage are different depending on the data type.\nData stored under the \"model\" directory include the best data produced by the numerical models (atmospheric or oceanic) by different external providers. Typically, these data are produced assimilating observational data in order to improve the algorithm functionalities which drive the models. Spatial, temporal resolution and temporal coverage depend on the model considered. \nThe data stored under the \"rolling\" section exist no longer than 15 days, meaning that they are permanently removed from the archive after that time period since they are superseded by better quality data, e.g., updated forecasts. The \"rolling\" archive includes also data subjected to further processing operations saved under the proper directories: the processing could include an interpolation to a finer grid in order to increase the spatial resolution (\"derived\" data), a subsampling by time or by space (\"generic\" data) or a transformation in a new format for specific purposes (\"dts\" data). The access to the data archive is allowed only via the Data Access Service described below.\n\nImplementation of the DAS-DTS modules \nFigure 3 shows a high-level representation of the DAS and its strict interaction with the DTS and the data archive, as implemented in the framework of the TESSA project. It is composed by totally automatized cyclic workflows that execute a set of steps on a set of data.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Interactions between the DAS, the DTS and the data archive\n\n\n\nThis interchange system has been implemented through some high-parameterized software components which schedule and parallelize the different jobs in a high-performance computing environment. The software modules developed for this system are the Downloader module, the DTS_organizer module and the Uploader module. Schematically, these modules download source raw data from a set of external providers and, after a set of transformations, taking into account the appropriate data format, store them in the data archive and publish them through the set of services exposed. Finally the Rolling_cleaner module is used to delete obsolete files and directories in the rolling archive. With the aim to increase the performance and the robustness of the system, all the mentioned components work in a multi-threaded environment: each thread is responsible for managing distinct sets of files. To cope with erroneous input, a MD5 checksum is also used. It is worth noting that the whole process is performed in a transparent way with respect to the services that act as consumers of the produced datasets in order to provide data always updated to the latest versions.\n\nThe Metadata Service and the metadata profile \nThe analysis of geospatial metadata standards has been the starting point for the definition of a specific metadata profile suitable for the description of the datasets produced in the framework of the TESSA project. ISO 19115, Geographic information \u2013 Metadata,\" is a standard of the International Organization for Standardization. It is a constituent of the ISO series 19100 standards for spatial metadata and defines the schema required for describing geographic information. In particular, this standard provides information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference and distribution of digital geographic data. ISO 19115 is applicable to the cataloguing of datasets and the full description of datasets, geographic datasets, dataset series and individual features and feature properties.\nThe ISO 19115 standard provides approximately 400 metadata elements, most of which are listed as \"optional.\" However, it has also drawn up a \"core metadata set for geographic dataset\" for search, access, transfer and exchange of geographic data; metadata profiles compliant to this international standard shall include this core to ensure homogeneity and interoperability. All metadata profiles based on ISO 19115 should include this minimum set for conformance with this international standard.\nThe ISO 19115 metadata standard has been widely adopted by a large number of international oceanographic and marine projects, such as the International Hydrographic Organization[35] and SeaDataNet. For its characteristics, it has been considered a valid solution also for the TESSA project, as capable of satisfying requirements of simplicity and interoperability. Thus, a specific metadata profile has been designed for describing and cataloging TESSA products, according to the requirements of the international metadata standard ISO 19115 and its XML implementation ISO 19139.\nA detailed analysis of TESSA datasets led to the definition of the key aspects about how, when and by whom a specific set of data was collected, how the data are accessed and managed, and which data formats are employed in order to obtain a general overview of data and metadata available. On the basis of this information, a minimum set of metadata has been defined as able to describe the datasets produced in the framework of the TESSA project. This schema includes all mandatory elements of the core metadata set for geographic datasets drawn up by the ISO 19115 standard for search, access, transfer and exchange of geographic data: dataset title, dataset reference date, abstract describing the dataset, dataset language, dataset topic category, metadata point of contact, metadata date stamp and online resource.\nThe ISO 19115 standard for metadata is being adopted internationally, as it provides a built-in mechanism to develop a \"community profile\" by extending the minimum set with additional metadata elements to suit the needs of a particular group of users.\nHence the generation of the specific TESSA metadata profile composed of the minimum set of metadata required to serve the full range of metadata applications (discovery and access) and another three optional elements that allow for a more extensive description and, at the same time, highlight the main features of TESSA products. \nAnother important step in the design of the TESSA Metadata Service has been the analysis of the information systems most widely adopted, at the international level, to support the activities of search and discovery.\nIn particular, the ESGF has been recognized and selected as the leading infrastructure for the management and access of large distributed data volumes for climate change research. It consists of a system of distributed and federated data nodes that interact dynamically through a peer-to-peer paradigm to handle climate science data, with multiple petabytes of disparate climate datasets from a lot of modeling efforts, such as CMIP5 and CORDEX. Internally, each ESGF node is composed of a set of services and applications for secure data\/metadata publication and access and user management. Moreover, the search service offers an intuitive tool based on search categories or \"facets\" (e.g., \"model\" or \"project\"), through which users are enabled to constrain their search results.\nFor these features, ESGF has been selected as the most appropriate information system for TESSA data dissemination and, above all, its visibility in the climate community.\n\nImplementation of the Metadata Service \nIn this phase, the metadata profiles for the description of datasets produced by TESSA models have been formalized; moreover, the installation of a dedicated ESGF data node has been performed, customized on the requirements and specificities of the TESSA project.\nFor its characteristics of simplicity and interoperability, the GeoNetwork metadata editor[36] has been chosen as the metadata editing tool for the description of datasets produced by TESSA models. In fact, GeoNetwork opensource is a multi-platform metadata catalogue application, designed to connect scientific communities and their data through the web environment. This software provides an easy-to-use web interface to search geospatial data across multiple catalogs, to publish geospatial data using the online metadata editing tools, to manage user and group accounts and to schedule metadata harvesting from other catalogs. In particular, GeoNetwork provides a powerful metadata editing tool that supports the ISO 19115 metadata profile and provides automatic metadata editing and management. Profiles can take the form of templates that are possible to fill in by using the metadata editor. Once the metadata profile has been compiled, it is also be possible to generate the related XML schema compliant with ISO 19139, to link data to related metadata description and to set the privileges to view metadata and to download the data.\nAs an example, Fig. 4 reports an extract of the metadata profile compiled for the description of AFS simulation dataset.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Metadata profile for an example of AFS dataset\n\n\n\nThus, this metadata profile represents a very powerful but, at the same time, user-friendly tool for the activities of search and discovery of the data produced in the framework of the TESSA project. As stated before, the TESSA information system also includes a dedicated ESGF data node, installed and properly configured for the project requirements. First of all, the web interface has been customized by inserting the description of the project and other related information. However, the most important step has been the mapping of TESSA data properties onto the \"search categories or facets,\" through which users can navigate the archive. In particular, new facets such as \"spatial_resolution\" and \"spatial_domain\" have been defined in order to better fit the specificity of TESSA datasets. These new facets have been added to configuration files and are now available in the web interface of the TESSA data node to support data search. In parallel, specific global attributes have been inserted in all output files of TESSA models in order to provide the metadata values for the corresponding search facets. These features are schematically represented in Fig. 5.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. List of facets for AFS products as published in the ESGF data node for TESSA\n\n\n\nThe Complex Data Analysis Module \nThe CDAM component has been designed with the aim to optimize the DSS job submissions in a multi-channel (web or mobile) environment. This component exploits existing technologies, widely spread in client\u2013server architectures, in order to simplify the complexity of the overall system, increasing the flexibility of the involved modules and the separation of concerns.\nFigure 6 shows an overview of the workflow related to TESSA DSS and the components involved in the operational chain.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. DSS request submission and interactions with the CDAM\n\n\n\nThe actors of such a scenario are (i) the user, interacting with the system using his mobile device or his computer, (ii) the SSA platform and (iii) the CDAM, which exploits a cluster infrastructure for the job execution. \nThe user sends a request to the SSA platform by using a desktop or mobile application related to a specific DSS. The request contains a set of parameters that will be parsed by the SSA platform and sent to the CDAM through a secure communication channel. Upon receiving the request, the CDAM gateway is responsible for preparing the environment for the job submission and for sending the parameter to the underlying cluster infrastructure, which executes the DSS algorithm and, at the end, sends a message to the SSA platform with the result of the processing.\n\nThe CDAM gateway \nThe CDAM gateway is designated to be the entry point for the job submission. It is responsible for managing the incoming requests on the basis of the different DSS and for interacting with the underlying cluster infrastructure for the algorithm execution. The CDAM gateway module is constituted by two components, the web interface and the CDAM scheduler.\nThe web interface provides the SSA platform with a unique and uniform interface since every request is served in the same way. In order to secure the communication channel between the SSA platform and the CDAM gateway, two levels of security constraints have to be satisfied. The former concerns the authentication\/authorization method based on the basic authentication mechanism; the sender of the request (in this case the SSA platform) needs to authenticate itself by providing the proper username and password. Moreover, the transfer of such information is protected by the HTTP protocol with SSL encryption (HTTPS). The latter regards a specific OS firewall policy which enables incoming communications from well-known IP range.\nThe CDAM scheduler is composed of a series of modules, one for each DSS or service that has been implemented. In particular, three types of services have been defined: submission of a job related to a specific DSS, deletion of an active job, and implementation of an echo service to check if the job submission service is available and to evaluate the response time.\nThe main tasks of each DSS-specific module are:\n1. to check the parameters received from the web interface;\n2. to perform the setup of the environment on the cluster infrastructure, by creating the directories required for the algorithm execution; and\n3. to contact the execution host with the correct parameters.\nBoth web interface and CDAM scheduler modules are based on a logging mechanism for keeping track of all the operations performed and the errors that occurred during the process.\n\nThe CDAM launcher \nThe CDAM launcher is hosted on the entry point of the cluster infrastructure and has the responsibility of properly managing the DSS submission on the cluster.\nAs the CDAM scheduler, the launcher is composed of one module for each DSS or service and performs the following operations: \n\n- checks the parameters received from the scheduler;\n- prepares the files that will be used as input for the algorithm execution;\n- performs the job submission; and\n- sends the result of the processing to the SSA platform.\nIt is worth noting that the design and implementation of the CDAM stack is independent from the software modules designated to start the submission; indeed, it provides general interfaces exposing a remote submission service able to hide the HPC resources for concurrent execution of several jobs. In TESSA, such a task is performed by the SSA platform which manages the incoming requests from the users and interacts with the CDAM for the DSS algorithms execution.\n\nOperational activity: TESSA data platform use cases \nThe TESSA data platform developed within the TESSA project represents an innovative solution not only as it provides efficient access to the data but also, above all, exploits and integrates different technologies to fully satisfy any potential user\u2019s need at the operational level. In fact, the three main components of the platform (DAS, Metadata Service and CDAM) are not isolated blocks; they interact with each other, efficiently supporting a number of services such as the SeaConditions operational chain[37], the automatic publication of datasets on the data nodes ESGF and the SSA DSS.\n\nAutomatic publishing procedure on the ESGF data node \nAn automatic procedure for data publishing on the ESGF data node has been designed and implemented in order to publish updated products on the TESSA ESGF portal.\nThis procedure is based on the strict interaction between the DTS and the Metadata Service. First, DTS applies pre-processing on the input data that are mainly the output of the hydrodynamic sub-regional models and, specifically, daily simulations and forecasts for the Adriatic, Ionian and Tyrrhenian seas, with different spatial and temporal resolution. This step is fundamental to make data compliant for the publication on the ESGF data node and involves the insertion of specific global attributes to provide the metadata values for the corresponding search facets. Once the pre-processing of the input data has been performed, the DTS modules alert the DAS component to publish the involved datasets on the ESGF node.\nFigure 7 shows the mentioned procedure in terms of a workflow of operational tasks: the delivery of TESSA outputs (simulations and forecasts, hourly and daily) from the production host to the target host, the automatic pre-processing of datasets by DTS operations and the upload of the processed data to the target directory.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 7. Operational chain for ESGF data publication\n\n\n\nOn the Metadata Service side, a controller module performs the publication of the different types of data preprocessed by DTS; specifically, the controller is a daemonized module that once a day sets up the directories map-files\/bulletindate and runs in background the procedures for the publication of the outputs of the models.\n\nDTS automatic procedures for DSS and numerical models transformations \nThe DTS is also responsible for pre-processing the input data for the different DSS, a set of tools for supporting users in making decisions or managing emergency situations and suggesting a strategy of intervention, namely WITOIL, VISIR and EarlyWarning. Specifically, WITOIL predicts the transport and transformation of real or hypothetical oil spills in the Mediterranean Sea, VISIR suggests the best nautical routes in any weather\u2013marine condition and EarlyWarning manages alert in case of extreme events by providing a daily update on weather (wind, air temperature) and oceanographic conditions (wave height, sea level, water temperature). In detail, the raw data, downloaded using the previously described DAS routines, are supplied to the automated pre-processing chains of the DTS, responsible for applying a series of transformations on the data in order to suit the specific DSS input requirements. At the end of the transformation process, the system is able to manage the different on-demand DSS requests starting the related algorithms.\nAs an example, WITOIL requires files of temperature, currents and wind at sea surface; in particular, the marine raw data are subjected to a spatial subsampling, selecting only the first 10 levels of depth. For the computation of the least-time nautical route, performed by VISIR, every day a single dataset of 10-day forecasts is provided; in this particular case, the DTS concatenates significant wave height, peak wave period and wave direction fields. For EarlyWarning, the pre-processing procedure applied to the files is very simple and consists of a data unzipping to create a data block useful to the execution of the algorithm. The DTS is responsible for applying all the mentioned processing on the input data in order to let the different DSS operational chains more quickly and efficiently exploit the automation provided by the DAS. Figures 8 and 9 show a nautical route computed by VISIR and a sea level forecast for the city of Venice computed by the EarlyWarning DSS.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 8. A motorboat route computed by VISIR; the linechart below the map shows the safety constraints that are violated by the geodetic route (black dots on the map) but not by the optimal route (red dots).\n\n\n\n\n\n\n\n\n\n\n\n\n Figure 9. EarlyWarning forecast for the sea level of Venice\n\n\n\nThe DAS is also extensively used by the DTS automatic processes for activating sub-regional model production (i.e., AFS, AIFS and SANIFS). The DAS modules are responsible for periodically downloading the source raw data from the external data provider. Once input data are available, the operational forecasting system is activated by specific DTS modules in order to schedule and manage subregional model execution and to properly synchronize the pre- and post-processing operational chains. \nThe interaction between the DAS and DTS is also important for the automated production of maps, starting from the outputs of the sub-regional model AFS, AIFS and SANIFS by means of dedicated operational chains.\n\nSeaConditions operational chain \nConcerning the SeaConditions operational chain, the main aim of the DTS is to create the properly processed and formatted data to visualize five days of weather\u2013oceanographic forecasts with three hours of temporal resolution for the first two and a half days and six hours for the others.\nDedicated DAS modules, day by day, download the source raw data from the external providers, storing them into the data archive. Raw data are then provided to the SeaConditions operational chains managed by DTS; at the end post-processed outputs are moved to an FTP server. SSA platform procedures are responsible for downloading them as input for the creation of the maps published on the SeaConditions web portal.\nDedicated DAS modules, day by day, download the source raw data from the external providers storing them into the data archive. Raw data are then provided to the SeaConditions operational chains managed by DTS; at the end post-processed outputs are moved to an FTP server.\nThis operational chain is based on a workflow manager implemented by a daemonized routine called DTS_SCHEDULER. This daemon manages the parallelization of the data processing tasks using stage-in and stage-out folders as interchange layers with the data archive. The different DTS production phases are represented in Fig. 10, including:\n\n- pre-elaboration phase, which is involved in the data splitting by time and by variables;\n- statistic phase, which aims to compute some statistics as minimum, maximum, mean and standard deviation;\n- interpolation phase, in which the split data resulting from the first phase are filtered by time and some are interpolated in order to increase their spatial resolution; and\n- compression phase, in which the data resulting from the interpolation phase are compressed: Lempel\u2013Ziv deflation, NetCDF4 packing algorithm (from the typical NC FLOAT to NC SHORT reducing the size by about 50 percent) and file zipping with gzip are operations that generally lead to a 70 to 90 percent reduction in file size. If the original NetCDF file occupies 1 GB, at the end of the packing and zipping process it will occupy approximately 100 MB.\nConclusions \nOne of the main objectives of the TESSA project was to develop a set of operational oceanographic services for the SSA. To reach this goal, advanced technological and software solutions were used as the backbone for the real time dissemination of the oceanographic data to a wide range of users in the maritime, tourism and scientific sectors. In this framework, the relevance of the TESSA data platform is apparent through its capacity to provide efficient and secure data access and strong support to high-level services, such as the DSS.\nAs stated in the introduction, the employment of well-known standards concerning protocols, services and output formats satisfies the FAIR guiding principles for scientific data management and stewardship. Specifically, the designed data platform relies on THREDDS for the publication of the TESSA final products (findable and accessible), while the OPeNDAP service exposes a standard DAP interface (interoperable). In addition, the ISO 19115 standard has been adopted for the definition of a metadata profile containing the mandatory elements of the core metadata set for geographic datasets (interoperable and reusable), and the search and discovery of data are supported by the ESGF data node and by the GeoNetwork information system (findable and accessible). Finally, access to the DSS methods is provided by a machine-readable interface exploiting the JSON format (accessible).\nIn general, the main components of this platform have been designed and developed by taking into account and satisfying a large number of additional requirements such as transparency, robustness, efficiency, fault tolerance and security. Moreover, it is important to emphasize that this platform represents a unique and innovative example of how different components interact with each other to support operational services for the SSA, such as SeaConditions and the other DSS (VISIR, WITOIL, Ocean-SAR, EarlyWarning).\nTherefore, these features make the TESSA data platform a valid prototype easily adopted to provide an efficient dissemination of maritime data and a consolidation of the management of operational oceanographic activities.\n\nData availability \nData of subregional models are periodically produced, exploiting the data platform operational chains and published at http:\/\/tds.tessa.cmcc.it.\n\nAcknowledgements \nThis research was funded by the project TESSA (Technologies for the Situational Sea Awareness; PON01_02823\/2) funded by the Italian Ministry for Education and Research (MIUR).\n\nCompeting interests \nThe authors declare that they have no conflict of interest.\n\nReferences \n\n\n\u2191 De Dominicis, M.; Pinardi, N.; Zodiatis, G.; Lardner, R. (2013). \"MEDSLIK-II, a Lagrangian marine surface oil spill model for short-term forecasting \u2013 Part 1: Theory\". Geoscientific Model Development 6 (6): 1851-1869. doi:10.5194\/gmd-6-1851-2013.   \n\n\u2191 De Dominicis, M.; Pinardi, N.; Zodiatis, G.; Archetti, R. (2013). \"MEDSLIK-II, a Lagrangian marine surface oil spill model for short-term forecasting \u2013 Part 2: Numerical simulations and validations\". Geoscientific Model Development 6 (6): 1871-1888. doi:10.5194\/gmd-6-1871-2013.   \n\n\u2191 Coppini, G.; Jansen, E.; Turrisi, G. et al. (2016). \"A new search-and-rescue service in the Mediterranean Sea: A demonstration of the operational capability and an evaluation of its performance using real case scenarios\". Natural Hazards and Earth System Sciences 16 (12): 2713-2727. doi:10.5194\/nhess-16-2713-2016.   \n\n\u2191 Mannarini, G.; Pinardi, N.; Cippini, G. et al. (2016). \"VISIR-I: Small vessels \u2013 Least-time nautical routes using wave forecasts\". Geoscientific Model Development 9 (4): 1597-1625. doi:10.5194\/gmd-9-1597-2016.   \n\n\u2191 Mannarini, G.; Turrisi, G.; D'Anca, A. et al. (2016). \"VISIR: Technological infrastructure of an operational service for safe and efficient navigation in the Mediterranean Sea\". Natural Hazards and Earth System Sciences 16 (8): 1791-1806. doi:10.5194\/nhess-16-1791-2016.   \n\n\u2191 Lecci, R.; Coppini, G.; Creti, S. et al. (2015). \"SeaConditions: Present and future sea conditions for safer navigation\". Proceedings from OCEANS 2015 - Genova 2015. doi:10.1109\/OCEANS-Genova.2015.7271764.   \n\n\u2191 Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J. et al. (2016). \"The FAIR Guiding Principles for scientific data management and stewardship\". Scientific Data 3: 160018. doi:10.1038\/sdata.2016.18. PMC PMC4792175. PMID 26978244. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175 .   \n\n\u2191 \"OGC network Common Data Form (netCDF) standards suite\". Open Geospatial Consortium. http:\/\/www.opengeospatial.org\/standards\/netcdf . Retrieved 30 January 2017 .   \n\n\u2191 \"Network Common Data Form (NetCDF)\". Unidata. University Corporation for Atmospheric Research. https:\/\/www.unidata.ucar.edu\/software\/netcdf\/ . Retrieved 30 January 2017 .   \n\n\u2191 \"Copernicus Marine Environment Monitoring Service\". European Commission. http:\/\/marine.copernicus.eu\/ . Retrieved 30 January 2017 .   \n\n\u2191 Bahurel, P.; Adragna, F.; Bell, M.J. et al. (2009). \"Ocean Monitoring and Forecasting Core Services, the European MyOcean Example\". Proceedings of OceanObs'09: Sustained Ocean Observations and Information for Society 2009. doi:10.5270\/OceanObs09.pp.02.   \n\n\u2191 \"Copernicus\". European Commission. http:\/\/www.copernicus.eu\/ . Retrieved 30 January 2017 .   \n\n\u2191 ISO\/TC 211 Geographic information\/Geomatics (May 2003). \"ISO 19115:2003 Geographic information -- Metadata\". International Organization for Standardization. https:\/\/www.iso.org\/standard\/26020.html .   \n\n\u2191 ISO\/TC 211 Geographic information\/Geomatics (April 2007). \"ISO\/TS 19139:2007 Geographic information -- Metadata -- XML schema implementation\". International Organization for Standardization. https:\/\/www.iso.org\/standard\/32557.html .   \n\n\u2191 \"INSPIRE: Infrastructure for Spatial Information in Europe\". European Commission. http:\/\/inspire.ec.europa.eu\/ . Retrieved 30 January 2017 .   \n\n\u2191 European Parliament (2007). \"Directive 2007\/2\/EC of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in the European Community (INSPIRE)\". Official Journal of the European Union L108 (25.4.2007). http:\/\/data.europa.eu\/eli\/dir\/2007\/2\/oj .   \n\n\u2191 Cinquini, L.; Crichton, D.; Mattmann, C. et al. (2014). \"The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data\". Future Generation Computer Systems 36 (July 2014): 400\u2013417. doi:10.1016\/j.future.2013.07.002.   \n\n\u2191 \"Earth System Grid Federation\". ESGF Executive Committee. https:\/\/esgf.llnl.gov\/ . Retrieved 30 January 2017 .   \n\n\u2191 \"EMODnet: European Marine Observation and Data Network\". European Commission. http:\/\/www.emodnet-mediterranean.eu\/ . Retrieved 30 January 2017 .   \n\n\u2191 Moussat, E.; Pinardi, N.; Manzella, G. et al. (2016). \"EMODnet MedSea Checkpoint for sustainable Blue Growth\". Geophysical Research Abstracts 18: 9103. http:\/\/meetingorganizer.copernicus.org\/EGU2016\/EGU2016-9103.pdf .   \n\n\u2191 \"SeaDataNet: Pan-European Infrastructure for Ocean & Marine Data Management\". European Commission. https:\/\/www.seadatanet.org\/ . Retrieved 30 January 2017 .   \n\n\u2191 \"CLIPC: Climate Information Portal\". European Commission. http:\/\/www.clipc.eu\/home . Retrieved 30 January 2017 .   \n\n\u2191 \"Standards Guide: ISO\/TC 211 Geographic information\/Geomatics\" (PDF). International Organization for Standardization. 01 June 2009. http:\/\/www.isotc211.org\/Outreach\/ISO_TC_211_Standards_Guide.pdf . Retrieved 30 January 2017 .   \n\n\u2191 Scalas, M.; Palmalisa, M.; Tedesco, L. et al. (2017). \"TESSA: Design and implementation of a platform for situational sea awareness\". Natural Hazards and Earth System Sciences 17 (2): 185-196. doi:10.5194\/nhess-17-185-2017.   \n\n\u2191 \"THREDDS Data Server (TDS)\". Unidata. University Corporation for Atmospheric Research. http:\/\/www.unidata.ucar.edu\/software\/thredds\/current\/tds\/ . Retrieved 30 January 2017 .   \n\n\u2191 Guarnieri, A.; Oddo, P.; Pastore, M.; Pinardi, N. (2008). \"The Adriatic Basin Forecasting System: New model and system development\". Proceeding of 5th EuroGOOS Conference, Exeter 2008. http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.584.9289&rep=rep1&type=pdf .   \n\n\u2191 Oddo, P.; Pinardi, N.; Zavatarelli, M. (2005). \"A numerical study of the interannual variability of the Adriatic Sea (2000\u20132002)\". Science of The Total Environment 353 (1\u20133): 39\u201356. doi:10.1016\/j.scitotenv.2005.09.061. PMID 16257438.   \n\n\u2191 Oddo, P.; Pinardi, N.; Zavatarelli, M.; Coluccelli, A. (2006). \"The Adriatic Basin forecasting system\". Acta Adriatica 47 (Sup. 1): 169\u2013184. http:\/\/hrcak.srce.hr\/8550 .   \n\n\u2191 Madec, G. (2008). \"NEMO Ocean General Circulation Model Reference Manual, Internal Report\". LODYC\/IPSL. http:\/\/www.nemo-ocean.eu\/About-NEMO\/Reference-manuals .   \n\n\u2191 Oddo, P.; Adani, M.; Pinardi, N. et al. (2009). \"A nested Atlantic-Mediterranean Sea general circulation model for operational forecasting\". Ocean Science 5 (4): 461-473. doi:10.5194\/os-5-461-2009.   \n\n\u2191 Federico, I.; Pinardi, N.; Coppini, G. et al. (2017). \"Coastal ocean forecasting with an unstructured grid model in the southern Adriatic and northern Ionian seas\". Natural Hazards and Earth System Sciences 17 (1): 45\u201359. doi:10.5194\/nhess-17-45-2017.   \n\n\u2191 de la Beaujardiere, J., ed. (15 March 2006). \"OpenGIS Web Map Server Implementation Specification\". Open Geospatial Consortium, Inc. http:\/\/portal.opengeospatial.org\/files\/?artifact_id=14416 . Retrieved 08 February 2017 .   \n\n\u2191 \"About OPeNDAP\". OPeNDAP. https:\/\/www.opendap.org\/about . Retrieved 30 January 2017 .   \n\n\u2191 \"GPFS: A Shared-Disk File System for Large Computing Clusters\". Proceedings of the 1st USENIX Conference on File and Storage Technologies 2002: 19. 2002. http:\/\/dl.acm.org\/citation.cfm?id=1083323.1083349 .   \n\n\u2191 \"International Hydrographic Organization\". International Hydrographic Organization. http:\/\/www.iho.int\/srv1 . Retrieved 30 January 2017 .   \n\n\u2191 \"GeoNetwork opensource\". Open Source Geospatial Foundation. http:\/\/geonetwork-opensource.org\/ . Retrieved 30 January 2017 .   \n\n\u2191 \"SeaConditions\". Links Management and Technology S.p.a. http:\/\/www.sea-conditions.com\/en\/ . Retrieved 30 January 2017 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. References in the original work were alphabetical; this website organizes references in order of appearance, by design. Grammar was cleaned up in several places. A URL to Unidata's NetCDF page was originally incorrect and updated for this version. URLs were added to the ISO and other standards references for improved reference information.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\">https:\/\/www.limswiki.org\/index.php\/Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on geoinformaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 26 April 2017, at 18:35.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,197 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","e86f9fcd9da316970f965a883c42e462_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_A_multi-service_data_management_platform_for_scientific_oceanographic_products skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:A multi-service data management platform for scientific oceanographic products<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>An efficient, secure and interoperable data platform solution has been developed in the TESSA project to provide fast navigation and access to the data stored in the data archive, as well as a standard-based metadata management support. The platform mainly targets scientific users and the situational sea awareness high-level services such as the decision support systems (DSS). These datasets are accessible through the following three main components: the Data Access Service (DAS), the Metadata Service and the Complex Data Analysis Module (CDAM). The DAS allows access to data stored in the archive by providing interfaces for different protocols and services for downloading, variable selection, data subsetting or map generation. Metadata Service is the heart of the information system of TESSA products and completes the overall infrastructure for data and metadata management. This component enables data search and discovery and addresses interoperability by exploiting widely adopted standards for geospatial data. Finally, the CDAM represents the back end of the TESSA DSS by performing on-demand complex data analysis tasks.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>TESSA (Development of TEchnology for Situational Sea Awareness) is a research project born from the collaboration between operational oceanography research and scientific computing groups in order to strengthen operational oceanography capabilities in Southern Italy for use by end users in the maritime, tourism and environmental protection sectors. This project has been very innovative as it has provided the integration of marine and ocean forecasts and analyses with advanced technological platforms. Specifically, an efficient, secure and interoperable data platform solution has been developed to provide fast navigation and access to the data stored in the data archive, as well as standard-based metadata management support. \n<\/p><p>This platform mainly targets scientific users and contains a set of high-level services such as the decision support systems (DSS) for supporting end users in managing emergency situations due to natural hazards in the Mediterranean Sea. For example, the DSS WITOIL<sup id=\"rdp-ebb-cite_ref-DeDominicisMEDSLIK-P113_1-0\" class=\"reference\"><a href=\"#cite_note-DeDominicisMEDSLIK-P113-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeDominicisMEDSLIK-P213_2-0\" class=\"reference\"><a href=\"#cite_note-DeDominicisMEDSLIK-P213-2\" rel=\"external_link\">[2]<\/a><\/sup> is crucial for oil spill accidents which could have severe impacts on the Mediterranean Basin and contribute effectively to the reduction of natural disaster risks. Moreover, the DSS Ocean-SAR<sup id=\"rdp-ebb-cite_ref-CoppiniANew16_3-0\" class=\"reference\"><a href=\"#cite_note-CoppiniANew16-3\" rel=\"external_link\">[3]<\/a><\/sup> supports the search-and-rescue (SAR) operations following accidents, and EarlyWarning manages alerts in cases of extreme events by providing near-real-time <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> on weather and oceanographic conditions. Finally, VISIR<sup id=\"rdp-ebb-cite_ref-MannariniVISIR-I16_4-0\" class=\"reference\"><a href=\"#cite_note-MannariniVISIR-I16-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MannariniVISIR16_5-0\" class=\"reference\"><a href=\"#cite_note-MannariniVISIR16-5\" rel=\"external_link\">[5]<\/a><\/sup> is able to compute the optimal ship route with the aim of increasing safety and efficiency of navigation. It relies on various forecast environmental fields which affect the vessel's stability in order to ensure that the computed route does not result in an exposure to dynamical hazards. In this context, the developed platform faces the lack of an efficient dissemination of marine environmental data in order to support situational sea awareness (SSA), which is strategically important for maritime safety and security.<sup id=\"rdp-ebb-cite_ref-LecciSea15_6-0\" class=\"reference\"><a href=\"#cite_note-LecciSea15-6\" rel=\"external_link\">[6]<\/a><\/sup> In fact, an updated situation awareness requires an advanced technological system to make data available for decision makers, improving the capacity of intervention and avoiding potential damages.\n<\/p><p>In a \"data-centric\" perspective \u2014 in which different services, applications or users make use of the outputs of regional or global numerical models \u2014 the TESSA data platform meets the request of near-real-time access to heterogeneous data with different accuracy, resolution or degrees of aggregation. The design phase has been driven by multiple needs that the developed solution had to satisfy. First of all, nowadays each data management system should satisfy the FAIR guiding principles for scientific data management and stewardship<sup id=\"rdp-ebb-cite_ref-WilkinsonTheFAIR16_7-0\" class=\"reference\"><a href=\"#cite_note-WilkinsonTheFAIR16-7\" rel=\"external_link\">[7]<\/a><\/sup>: they face the lack of a shared, comprehensive, broadly applicable and articulated guideline concerning the publication of scientific data. Specifically, they identify findability, accessibility, interoperability and reusability as the four principles which need to be addressed by data (and associated metadata) to be considered a good publication. As explained later, the employment of well-known standards concerning protocols, services and output formats satisfies these requirements. In addition, the need for a service that provides information about sea conditions 24\/7 at high and very high spatial and temporal resolution has been addressed by exploiting high-performance and high-availability hardware and software solutions. The developed platform must be able to support the requests of intermediate and common users. To this end, data must be available in the native and standard format (NetCDF)<sup id=\"rdp-ebb-cite_ref-OGCnetCDF_8-0\" class=\"reference\"><a href=\"#cite_note-OGCnetCDF-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-UnidataNetCDF_9-0\" class=\"reference\"><a href=\"#cite_note-UnidataNetCDF-9\" rel=\"external_link\">[9]<\/a><\/sup> as output of the oceanographic models, through a simple and intuitive platform suitable for machine-based interactions, in order to feed user-friendly services for displaying clear maps and graphs. At the end, the system has to provide on-demand services to support decisions; the users must be able to interact with the datasets produced by the models in near-real time. As such, the platform has to provide services and datasets suitable for on-demand processing while minimizing the downloading time and the related input file size.\n<\/p><p>This paper is organized as follows: The next section presents a survey on related work, and in the following section \"The data platform architecture\" an architectural overview of the main data platform components is provided. The focus is on providing easier and unified access to the heterogeneous data produced in the framework of the TESSA project. The implemented data archive and the modules that make the datasets available to the entire set of services are presented in \"The Data Access Service.\" In \"The Metadata Service and the metadata profile,\" the Metadata Service features are detailed from a methodological and technical point-of-view. The module developed in order to serve remote DSS submission requests is described in \"The Complex Data Analysis Module.\" Finally in the final section on operational activity, a few use cases are presented, highlighting the operational chains which exploit the data platform services.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Related_work\">Related work<\/span><\/h2>\n<p>The need for marine and oceanic data management supporting the SSA and the operational oceanography has led to the definition and development of various platforms that provide different types of data and services. In this context, the MyOcean project<sup id=\"rdp-ebb-cite_ref-CopernicusMEMS_10-0\" class=\"reference\"><a href=\"#cite_note-CopernicusMEMS-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BahurelOcean09_11-0\" class=\"reference\"><a href=\"#cite_note-BahurelOcean09-11\" rel=\"external_link\">[11]<\/a><\/sup> can be mentioned, as this project implementation was in line with the best practices of the GMES\/Copernicus framework.<sup id=\"rdp-ebb-cite_ref-Copernicus_12-0\" class=\"reference\"><a href=\"#cite_note-Copernicus-12\" rel=\"external_link\">[12]<\/a><\/sup>\n<\/p><p>Specifically, regarding data access, MyOcean provides the scientific community with a unified interface designed to take into account various international standards (ISO 19115<sup id=\"rdp-ebb-cite_ref-ISO19115:2003_13-0\" class=\"reference\"><a href=\"#cite_note-ISO19115:2003-13\" rel=\"external_link\">[13]<\/a><\/sup>, ISO 19139<sup id=\"rdp-ebb-cite_ref-ISO19139:2007_14-0\" class=\"reference\"><a href=\"#cite_note-ISO19139:2007-14\" rel=\"external_link\">[14]<\/a><\/sup> and INSPIRE<sup id=\"rdp-ebb-cite_ref-INSPIRE_15-0\" class=\"reference\"><a href=\"#cite_note-INSPIRE-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-EuropeanDirective07_16-0\" class=\"reference\"><a href=\"#cite_note-EuropeanDirective07-16\" rel=\"external_link\">[16]<\/a><\/sup>). Concerning data management, MyOcean relies on OPeNDAP\/THREDDS for tasks like map subsetting and FTP for direct download. In addition, a valid solution is represented by the Earth System Grid Federation (ESGF)<sup id=\"rdp-ebb-cite_ref-CinquiniTheEarth14_17-0\" class=\"reference\"><a href=\"#cite_note-CinquiniTheEarth14-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ESGF_18-0\" class=\"reference\"><a href=\"#cite_note-ESGF-18\" rel=\"external_link\">[18]<\/a><\/sup>, a federated system used as metadata service with advanced features, that will be described later. EMODNET MEDSEA Checkpoint<sup id=\"rdp-ebb-cite_ref-EMODnet_19-0\" class=\"reference\"><a href=\"#cite_note-EMODnet-19\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MoussatEMODnet16_20-0\" class=\"reference\"><a href=\"#cite_note-MoussatEMODnet16-20\" rel=\"external_link\">[20]<\/a><\/sup> is another solution supporting data collection and data search and discovery: it exploits a checkpoint browser and a checkpoint dashboard, which presents indicators automatically produced from information databases. SeaDataNet<sup id=\"rdp-ebb-cite_ref-SeaDataNet_21-0\" class=\"reference\"><a href=\"#cite_note-SeaDataNet-21\" rel=\"external_link\">[21]<\/a><\/sup> represents a distributed marine data management infrastructure able to manage different large datasets related to in situ and remote observation of the seas and oceans. Through a distributed network approach, it provides an integrated overview and access to datasets provided by 90 national oceanographic and marine data centers. Finally, another efficient solution has been developed within the project CLIPC.<sup id=\"rdp-ebb-cite_ref-CLIPC_22-0\" class=\"reference\"><a href=\"#cite_note-CLIPC-22\" rel=\"external_link\">[22]<\/a><\/sup> It provides access to climate information, including data from satellite and in situ observations, transformed data products and climate change impact indicators. Moreover, users can exploit the CLIPC toolbox to generate, compare, manipulate and combine indicators, create a user basket or launch new jobs for indices calculation.\n<\/p><p>These systems support a wide range of functionalities such as data access, classification, search and discovery and downloading by means of a distributed architecture. Graphs and maps production are also supported, and some of these propose tools to manipulate and combine datasets. The proposed architecture of the TESSA data platform should offer superior and fast data management functionalities, producing datasets suitable for serving different kinds of services in near-real time; access, search and discovery are just some of the features offered. The developed system, in fact, provides a common interface for submitting complex algorithms in a high-performance computing environment using standard approaches and formats exploiting a unified infrastructure for supporting different scenarios and applications.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_data_platform_architecture\">The data platform architecture<\/span><\/h2>\n<p>As seen in Fig. 1, the TESSA data platform is composed of three main components: the Data Access Service (DAS), the Metadata Service and the Complex Data Analysis Module (CDAM).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"4b4dead4f347d25d68c7a683463de850\"><img alt=\"Fig1 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/e\/e2\/Fig1_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> The TESSA data platform<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The DAS allows the access to the data archive, the storage of the data produced in the framework of the TESSA project. The main consumers of these services are the \u201cintermediate users\u201d (the scientific community) and the automatic procedures fed by produced datasets to perform the needed transformations. In this context, the DAS represents the single interface to access the data archive of TESSA; its design has been driven by the aim to satisfy requirements such as transparency, robustness, efficiency, fault tolerance, security, data delivery, subsetting and browsing. This implies that this component must necessarily be \"data-centric\" and \"multi-service\" in order to offer access to the data, often stored in a single instance on the data archive, through different protocols and, therefore, different levels of functionality.\n<\/p><p>In addition, the need of interoperability with other data management platforms has driven the design toward the inclusion of well-known services:\n<\/p>\n<dl><dd>\u2013 \"domain-oriented,\" able to provide specific functionalities for the climate change domain; and<\/dd><\/dl>\n<dl><dd>\u2013 \"general purpose,\" to satisfy general requirements exploiting common services and protocols.<\/dd><\/dl>\n<p>The Metadata Service has been designed as the heart of the information system of the TESSA products, to support the activities of search and discovery and metadata management. Moreover, the Metadata Service addresses the requirements of interoperability, considered a key feature in order to provide to any potential user easier and unified access to the different types of data produced by the project. These requirements have been satisfied through the adoption of a metadata profile compliant to the ISO standards for geospatial data<sup id=\"rdp-ebb-cite_ref-ISOTC2011StandardsGuide_23-0\" class=\"reference\"><a href=\"#cite_note-ISOTC2011StandardsGuide-23\" rel=\"external_link\">[23]<\/a><\/sup> and the implementation of an efficient information system relying on a specific ESGF data node\n<\/p><p>Finally, the CDAM supports the SSA high-level services by performing on-demand complex analysis on the datasets stored in the data archive. In particular, a number of intelligent decisions support systems for complex dynamical environments have been designed and developed, exploiting high-performance computing functionalities to execute their algorithms. One of the goals is to deploy an infrastructure where a DSS requires the execution of a complex task including (i) accessing data stored in the archive, (ii) executing one or more preliminary tasks for processing them and (iii) storing the results in a specific section of the data archive or publishing them on other sites. Each DSS request interacts with the platform involving a three-layer infrastructure: the user client (web or mobile platform), the SSA platform<sup id=\"rdp-ebb-cite_ref-ScalasTESSA17_24-0\" class=\"reference\"><a href=\"#cite_note-ScalasTESSA17-24\" rel=\"external_link\">[24]<\/a><\/sup> and the CDAM. Each layer communicates with their linked tiers exchanging messages using well-known protocols, and it has been developed in order to implement a high separation of concerns.\n<\/p><p>The design and the main features of these modules will be detailed in the following sections.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_Data_Access_Service\">The Data Access Service<\/span><\/h2>\n<p>As stated in the previous section, the DAS represents the single interface to access the data archive of TESSA; specifically, it has been designed with the aim to set up a multi-service platform to access and analyze scientific data. Among the exposed services, it is important to mention the THREDDS Data Server (TDS)<sup id=\"rdp-ebb-cite_ref-THREDDS_25-0\" class=\"reference\"><a href=\"#cite_note-THREDDS-25\" rel=\"external_link\">[25]<\/a><\/sup>, which represents a web server that provides metadata and data access for scientific datasets, using a variety of remote data access protocols. More in detail, it offers a set of services that allows users direct access to data produced in the context of the TESSA project, namely the output of the AFS (Adriatic Forecasting System)<sup id=\"rdp-ebb-cite_ref-GuarnieriTheAdri08_26-0\" class=\"reference\"><a href=\"#cite_note-GuarnieriTheAdri08-26\" rel=\"external_link\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OddoANumer05_27-0\" class=\"reference\"><a href=\"#cite_note-OddoANumer05-27\" rel=\"external_link\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OddoTheAdri06_28-0\" class=\"reference\"><a href=\"#cite_note-OddoTheAdri06-28\" rel=\"external_link\">[28]<\/a><\/sup>, AIFS<sup id=\"rdp-ebb-cite_ref-MadecNEMOOcean08_29-0\" class=\"reference\"><a href=\"#cite_note-MadecNEMOOcean08-29\" rel=\"external_link\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OddoANested09_30-0\" class=\"reference\"><a href=\"#cite_note-OddoANested09-30\" rel=\"external_link\">[30]<\/a><\/sup> and SANIFS<sup id=\"rdp-ebb-cite_ref-FedericoCoastal17_31-0\" class=\"reference\"><a href=\"#cite_note-FedericoCoastal17-31\" rel=\"external_link\">[31]<\/a><\/sup> sub-regional models. In particular, the THREDDS includes the WMS (Web Map Service)<sup id=\"rdp-ebb-cite_ref-OGCOpenGIS06_32-0\" class=\"reference\"><a href=\"#cite_note-OGCOpenGIS06-32\" rel=\"external_link\">[32]<\/a><\/sup> for the dynamic generation of maps created from geographically referenced data and directly downloaded using the HTTP protocol. Furthermore, it includes the OPeNDAP<sup id=\"rdp-ebb-cite_ref-OPeNDAP_33-0\" class=\"reference\"><a href=\"#cite_note-OPeNDAP-33\" rel=\"external_link\">[33]<\/a><\/sup> service, which is a framework that aims to simplify the sharing of scientific information on the web by making available local data from remote connections, providing also variable selection and spatial and temporal subsetting capabilities. To improve performance in terms of load balancing and increase the fault tolerance of the system, a multiplexed configuration for the THREDDS installation based on two different hosts has been used. In addition, a sub-component, the Data Transformation Service (DTS), represents a middleware responsible for applying different data transformations in order to make data compliant to the different protocols offered by the DAS and therefore available to the data consumers (users or services) in a transparent way.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_dataset_repository:_The_TESSA_data_archive\">The dataset repository: The TESSA data archive<\/span><\/h3>\n<p>The data archive represents the storage for all the data produced within the TESSA project. It has been designed and implemented in order to ease the data search-and-discovery phase exploiting their classification while, from an infrastructure point of view, data are stored on a GPFS<sup id=\"rdp-ebb-cite_ref-SchmuckGPFS02_34-0\" class=\"reference\"><a href=\"#cite_note-SchmuckGPFS02-34\" rel=\"external_link\">[34]<\/a><\/sup> file system (in RAID 6 configuration), able to provide fast data access and a high level of fault tolerance. Figure 2 shows a high-level schema of the archive.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"c4af0179dc5bc61c355096d5c958704d\"><img alt=\"Fig2 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/e\/e5\/Fig2_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> High-level representation of the data archive<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The data can be subdivided into two categories:\n<\/p>\n<dl><dd>\u2013 historical data, which are permanently stored into the archive, including observations and model reanalysis and best analysis; and<\/dd><\/dl>\n<dl><dd>\u2013 rolling data, which are subjected to a rolling procedure, including model worst analysis, simulations and forecasts.<\/dd><\/dl>\n<p>Considering the different kinds of data, the data archive has been created following a tree-based topology where the directory names reflect the data classification (historical as first level, model or observation as second level, atmospheric or oceanographic as third level, etc.). The directory tree includes also the institute providing the data, the name of the system that produced the data and the temporal resolution of the data. This kind of classification eases and speeds up the search and discovery of data by intermediate\/scientific users and automatic procedures. Concerning the \"historical\" archive, data stored under \"obs\" include observations retrieved by different instruments (satellites, in situ buoys, vessels, etc.) and consider measurements of the main oceanographic variables (temperature, altimetry, salinity, chlorophyll concentration, etc.). The spatial and the temporal resolution and the temporal coverage are different depending on the data type.\n<\/p><p>Data stored under the \"model\" directory include the best data produced by the numerical models (atmospheric or oceanic) by different external providers. Typically, these data are produced assimilating observational data in order to improve the algorithm functionalities which drive the models. Spatial, temporal resolution and temporal coverage depend on the model considered. \n<\/p><p>The data stored under the \"rolling\" section exist no longer than 15 days, meaning that they are permanently removed from the archive after that time period since they are superseded by better quality data, e.g., updated forecasts. The \"rolling\" archive includes also data subjected to further processing operations saved under the proper directories: the processing could include an interpolation to a finer grid in order to increase the spatial resolution (\"derived\" data), a subsampling by time or by space (\"generic\" data) or a transformation in a new format for specific purposes (\"dts\" data). The access to the data archive is allowed only via the Data Access Service described below.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Implementation_of_the_DAS-DTS_modules\">Implementation of the DAS-DTS modules<\/span><\/h3>\n<p>Figure 3 shows a high-level representation of the DAS and its strict interaction with the DTS and the data archive, as implemented in the framework of the TESSA project. It is composed by totally automatized cyclic workflows that execute a set of steps on a set of data.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"80959ee1d1a47a57438ae8d883687027\"><img alt=\"Fig3 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/4\/43\/Fig3_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Interactions between the DAS, the DTS and the data archive<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>This interchange system has been implemented through some high-parameterized software components which schedule and parallelize the different jobs in a high-performance computing environment. The software modules developed for this system are the Downloader module, the DTS_organizer module and the Uploader module. Schematically, these modules download source raw data from a set of external providers and, after a set of transformations, taking into account the appropriate data format, store them in the data archive and publish them through the set of services exposed. Finally the Rolling_cleaner module is used to delete obsolete files and directories in the rolling archive. With the aim to increase the performance and the robustness of the system, all the mentioned components work in a multi-threaded environment: each thread is responsible for managing distinct sets of files. To cope with erroneous input, a MD5 checksum is also used. It is worth noting that the whole process is performed in a transparent way with respect to the services that act as consumers of the produced datasets in order to provide data always updated to the latest versions.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_Metadata_Service_and_the_metadata_profile\">The Metadata Service and the metadata profile<\/span><\/h2>\n<p>The analysis of geospatial metadata standards has been the starting point for the definition of a specific metadata profile suitable for the description of the datasets produced in the framework of the TESSA project. ISO 19115, Geographic information \u2013 Metadata,\" is a standard of the <a href=\"https:\/\/www.limswiki.org\/index.php\/International_Organization_for_Standardization\" title=\"International Organization for Standardization\" target=\"_blank\" class=\"wiki-link\" data-key=\"116defc5d89c8a55f5b7c1be0790b442\">International Organization for Standardization<\/a>. It is a constituent of the ISO series 19100 standards for spatial metadata and defines the schema required for describing geographic information. In particular, this standard provides information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference and distribution of digital geographic data. ISO 19115 is applicable to the cataloguing of datasets and the full description of datasets, geographic datasets, dataset series and individual features and feature properties.\n<\/p><p>The ISO 19115 standard provides approximately 400 metadata elements, most of which are listed as \"optional.\" However, it has also drawn up a \"core metadata set for geographic dataset\" for search, access, transfer and exchange of geographic data; metadata profiles compliant to this international standard shall include this core to ensure homogeneity and interoperability. All metadata profiles based on ISO 19115 should include this minimum set for conformance with this international standard.\n<\/p><p>The ISO 19115 metadata standard has been widely adopted by a large number of international oceanographic and marine projects, such as the International Hydrographic Organization<sup id=\"rdp-ebb-cite_ref-IHO_35-0\" class=\"reference\"><a href=\"#cite_note-IHO-35\" rel=\"external_link\">[35]<\/a><\/sup> and SeaDataNet. For its characteristics, it has been considered a valid solution also for the TESSA project, as capable of satisfying requirements of simplicity and interoperability. Thus, a specific metadata profile has been designed for describing and cataloging TESSA products, according to the requirements of the international metadata standard ISO 19115 and its XML implementation ISO 19139.\n<\/p><p>A detailed analysis of TESSA datasets led to the definition of the key aspects about how, when and by whom a specific set of data was collected, how the data are accessed and managed, and which data formats are employed in order to obtain a general overview of data and metadata available. On the basis of this information, a minimum set of metadata has been defined as able to describe the datasets produced in the framework of the TESSA project. This schema includes all mandatory elements of the core metadata set for geographic datasets drawn up by the ISO 19115 standard for search, access, transfer and exchange of geographic data: dataset title, dataset reference date, abstract describing the dataset, dataset language, dataset topic category, metadata point of contact, metadata date stamp and online resource.\n<\/p><p>The ISO 19115 standard for metadata is being adopted internationally, as it provides a built-in mechanism to develop a \"community profile\" by extending the minimum set with additional metadata elements to suit the needs of a particular group of users.\n<\/p><p>Hence the generation of the specific TESSA metadata profile composed of the minimum set of metadata required to serve the full range of metadata applications (discovery and access) and another three optional elements that allow for a more extensive description and, at the same time, highlight the main features of TESSA products. \n<\/p><p>Another important step in the design of the TESSA Metadata Service has been the analysis of the information systems most widely adopted, at the international level, to support the activities of search and discovery.\n<\/p><p>In particular, the ESGF has been recognized and selected as the leading infrastructure for the management and access of large distributed data volumes for climate change research. It consists of a system of distributed and federated data nodes that interact dynamically through a peer-to-peer paradigm to handle climate science data, with multiple petabytes of disparate climate datasets from a lot of modeling efforts, such as CMIP5 and CORDEX. Internally, each ESGF node is composed of a set of services and applications for secure data\/metadata publication and access and user management. Moreover, the search service offers an intuitive tool based on search categories or \"facets\" (e.g., \"model\" or \"project\"), through which users are enabled to constrain their search results.\n<\/p><p>For these features, ESGF has been selected as the most appropriate information system for TESSA data dissemination and, above all, its visibility in the climate community.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Implementation_of_the_Metadata_Service\">Implementation of the Metadata Service<\/span><\/h3>\n<p>In this phase, the metadata profiles for the description of datasets produced by TESSA models have been formalized; moreover, the installation of a dedicated ESGF data node has been performed, customized on the requirements and specificities of the TESSA project.\n<\/p><p>For its characteristics of simplicity and interoperability, the GeoNetwork metadata editor<sup id=\"rdp-ebb-cite_ref-GeoNetworkOS_36-0\" class=\"reference\"><a href=\"#cite_note-GeoNetworkOS-36\" rel=\"external_link\">[36]<\/a><\/sup> has been chosen as the metadata editing tool for the description of datasets produced by TESSA models. In fact, GeoNetwork opensource is a multi-platform metadata catalogue application, designed to connect scientific communities and their data through the web environment. This software provides an easy-to-use web interface to search geospatial data across multiple catalogs, to publish geospatial data using the online metadata editing tools, to manage user and group accounts and to schedule metadata harvesting from other catalogs. In particular, GeoNetwork provides a powerful metadata editing tool that supports the ISO 19115 metadata profile and provides automatic metadata editing and management. Profiles can take the form of templates that are possible to fill in by using the metadata editor. Once the metadata profile has been compiled, it is also be possible to generate the related XML schema compliant with ISO 19139, to link data to related metadata description and to set the privileges to view metadata and to download the data.\n<\/p><p>As an example, Fig. 4 reports an extract of the metadata profile compiled for the description of AFS simulation dataset.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"79bb92a8dc93b7db2a026b256edd86b6\"><img alt=\"Fig4 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/8\/8e\/Fig4_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Metadata profile for an example of AFS dataset<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Thus, this metadata profile represents a very powerful but, at the same time, user-friendly tool for the activities of search and discovery of the data produced in the framework of the TESSA project. As stated before, the TESSA information system also includes a dedicated ESGF data node, installed and properly configured for the project requirements. First of all, the web interface has been customized by inserting the description of the project and other related information. However, the most important step has been the mapping of TESSA data properties onto the \"search categories or facets,\" through which users can navigate the archive. In particular, new facets such as \"spatial_resolution\" and \"spatial_domain\" have been defined in order to better fit the specificity of TESSA datasets. These new facets have been added to configuration files and are now available in the web interface of the TESSA data node to support data search. In parallel, specific global attributes have been inserted in all output files of TESSA models in order to provide the metadata values for the corresponding search facets. These features are schematically represented in Fig. 5.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"f89798a3e12bcedb973d50a549f41150\"><img alt=\"Fig5 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/2\/29\/Fig5_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> List of facets for AFS products as published in the ESGF data node for TESSA<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"The_Complex_Data_Analysis_Module\">The Complex Data Analysis Module<\/span><\/h2>\n<p>The CDAM component has been designed with the aim to optimize the DSS job submissions in a multi-channel (web or mobile) environment. This component exploits existing technologies, widely spread in client\u2013server architectures, in order to simplify the complexity of the overall system, increasing the flexibility of the involved modules and the separation of concerns.\n<\/p><p>Figure 6 shows an overview of the workflow related to TESSA DSS and the components involved in the operational chain.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"b07707fea10b5586d24ae1ebb12470fb\"><img alt=\"Fig6 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/1\/11\/Fig6_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> DSS request submission and interactions with the CDAM<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The actors of such a scenario are (i) the user, interacting with the system using his mobile device or his computer, (ii) the SSA platform and (iii) the CDAM, which exploits a cluster infrastructure for the job execution. \n<\/p><p>The user sends a request to the SSA platform by using a desktop or mobile application related to a specific DSS. The request contains a set of parameters that will be parsed by the SSA platform and sent to the CDAM through a secure communication channel. Upon receiving the request, the CDAM gateway is responsible for preparing the environment for the job submission and for sending the parameter to the underlying cluster infrastructure, which executes the DSS algorithm and, at the end, sends a message to the SSA platform with the result of the processing.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_CDAM_gateway\">The CDAM gateway<\/span><\/h3>\n<p>The CDAM gateway is designated to be the entry point for the job submission. It is responsible for managing the incoming requests on the basis of the different DSS and for interacting with the underlying cluster infrastructure for the algorithm execution. The CDAM gateway module is constituted by two components, the web interface and the CDAM scheduler.\n<\/p><p>The web interface provides the SSA platform with a unique and uniform interface since every request is served in the same way. In order to secure the communication channel between the SSA platform and the CDAM gateway, two levels of security constraints have to be satisfied. The former concerns the authentication\/authorization method based on the basic authentication mechanism; the sender of the request (in this case the SSA platform) needs to authenticate itself by providing the proper username and password. Moreover, the transfer of such information is protected by the HTTP protocol with SSL encryption (HTTPS). The latter regards a specific OS firewall policy which enables incoming communications from well-known IP range.\n<\/p><p>The CDAM scheduler is composed of a series of modules, one for each DSS or service that has been implemented. In particular, three types of services have been defined: submission of a job related to a specific DSS, deletion of an active job, and implementation of an echo service to check if the job submission service is available and to evaluate the response time.\n<\/p><p>The main tasks of each DSS-specific module are:\n<\/p><p>1. to check the parameters received from the web interface;\n<\/p><p>2. to perform the setup of the environment on the cluster infrastructure, by creating the directories required for the algorithm execution; and\n<\/p><p>3. to contact the execution host with the correct parameters.\n<\/p><p>Both web interface and CDAM scheduler modules are based on a logging mechanism for keeping track of all the operations performed and the errors that occurred during the process.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_CDAM_launcher\">The CDAM launcher<\/span><\/h3>\n<p>The CDAM launcher is hosted on the entry point of the cluster infrastructure and has the responsibility of properly managing the DSS submission on the cluster.\n<\/p><p>As the CDAM scheduler, the launcher is composed of one module for each DSS or service and performs the following operations: \n<\/p>\n<dl><dd>- checks the parameters received from the scheduler;<\/dd><\/dl>\n<dl><dd>- prepares the files that will be used as input for the algorithm execution;<\/dd><\/dl>\n<dl><dd>- performs the job submission; and<\/dd><\/dl>\n<dl><dd>- sends the result of the processing to the SSA platform.<\/dd><\/dl>\n<p>It is worth noting that the design and implementation of the CDAM stack is independent from the software modules designated to start the submission; indeed, it provides general interfaces exposing a remote submission service able to hide the HPC resources for concurrent execution of several jobs. In TESSA, such a task is performed by the SSA platform which manages the incoming requests from the users and interacts with the CDAM for the DSS algorithms execution.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Operational_activity:_TESSA_data_platform_use_cases\">Operational activity: TESSA data platform use cases<\/span><\/h2>\n<p>The TESSA data platform developed within the TESSA project represents an innovative solution not only as it provides efficient access to the data but also, above all, exploits and integrates different technologies to fully satisfy any potential user\u2019s need at the operational level. In fact, the three main components of the platform (DAS, Metadata Service and CDAM) are not isolated blocks; they interact with each other, efficiently supporting a number of services such as the SeaConditions operational chain<sup id=\"rdp-ebb-cite_ref-SeaConditions_37-0\" class=\"reference\"><a href=\"#cite_note-SeaConditions-37\" rel=\"external_link\">[37]<\/a><\/sup>, the automatic publication of datasets on the data nodes ESGF and the SSA DSS.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Automatic_publishing_procedure_on_the_ESGF_data_node\">Automatic publishing procedure on the ESGF data node<\/span><\/h3>\n<p>An automatic procedure for data publishing on the ESGF data node has been designed and implemented in order to publish updated products on the TESSA ESGF portal.\n<\/p><p>This procedure is based on the strict interaction between the DTS and the Metadata Service. First, DTS applies pre-processing on the input data that are mainly the output of the hydrodynamic sub-regional models and, specifically, daily simulations and forecasts for the Adriatic, Ionian and Tyrrhenian seas, with different spatial and temporal resolution. This step is fundamental to make data compliant for the publication on the ESGF data node and involves the insertion of specific global attributes to provide the metadata values for the corresponding search facets. Once the pre-processing of the input data has been performed, the DTS modules alert the DAS component to publish the involved datasets on the ESGF node.\n<\/p><p>Figure 7 shows the mentioned procedure in terms of a workflow of operational tasks: the delivery of TESSA outputs (simulations and forecasts, hourly and daily) from the production host to the target host, the automatic pre-processing of datasets by DTS operations and the upload of the processed data to the target directory.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5ab7e1e3b695831f0288bd8abe159437\"><img alt=\"Fig7 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/a\/a8\/Fig7_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 7.<\/b> Operational chain for ESGF data publication<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>On the Metadata Service side, a controller module performs the publication of the different types of data preprocessed by DTS; specifically, the controller is a daemonized module that once a day sets up the directories <i>map-files\/bulletindate<\/i> and runs in background the procedures for the publication of the outputs of the models.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"DTS_automatic_procedures_for_DSS_and_numerical_models_transformations\">DTS automatic procedures for DSS and numerical models transformations<\/span><\/h3>\n<p>The DTS is also responsible for pre-processing the input data for the different DSS, a set of tools for supporting users in making decisions or managing emergency situations and suggesting a strategy of intervention, namely WITOIL, VISIR and EarlyWarning. Specifically, WITOIL predicts the transport and transformation of real or hypothetical oil spills in the Mediterranean Sea, VISIR suggests the best nautical routes in any weather\u2013marine condition and EarlyWarning manages alert in case of extreme events by providing a daily update on weather (wind, air temperature) and oceanographic conditions (wave height, sea level, water temperature). In detail, the raw data, downloaded using the previously described DAS routines, are supplied to the automated pre-processing chains of the DTS, responsible for applying a series of transformations on the data in order to suit the specific DSS input requirements. At the end of the transformation process, the system is able to manage the different on-demand DSS requests starting the related algorithms.\n<\/p><p>As an example, WITOIL requires files of temperature, currents and wind at sea surface; in particular, the marine raw data are subjected to a spatial subsampling, selecting only the first 10 levels of depth. For the computation of the least-time nautical route, performed by VISIR, every day a single dataset of 10-day forecasts is provided; in this particular case, the DTS concatenates significant wave height, peak wave period and wave direction fields. For EarlyWarning, the pre-processing procedure applied to the files is very simple and consists of a data unzipping to create a data block useful to the execution of the algorithm. The DTS is responsible for applying all the mentioned processing on the input data in order to let the different DSS operational chains more quickly and efficiently exploit the automation provided by the DAS. Figures 8 and 9 show a nautical route computed by VISIR and a sea level forecast for the city of Venice computed by the EarlyWarning DSS.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"7ed2410760fcf7cda6eef42a44f6bd09\"><img alt=\"Fig8 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/e\/e5\/Fig8_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 8.<\/b> A motorboat route computed by VISIR; the linechart below the map shows the safety constraints that are violated by the geodetic route (black dots on the map) but not by the optimal route (red dots).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig9_D%27Anca_NatHazEarth2017_17-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"1ad739fe3587fea4539a4dfe238e5e34\"><img alt=\"Fig9 D'Anca NatHazEarth2017 17-2.png\" src=\"https:\/\/www.limswiki.org\/images\/1\/15\/Fig9_D%27Anca_NatHazEarth2017_17-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 9.<\/b> EarlyWarning forecast for the sea level of Venice<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The DAS is also extensively used by the DTS automatic processes for activating sub-regional model production (i.e., AFS, AIFS and SANIFS). The DAS modules are responsible for periodically downloading the source raw data from the external data provider. Once input data are available, the operational forecasting system is activated by specific DTS modules in order to schedule and manage subregional model execution and to properly synchronize the pre- and post-processing operational chains. \n<\/p><p>The interaction between the DAS and DTS is also important for the automated production of maps, starting from the outputs of the sub-regional model AFS, AIFS and SANIFS by means of dedicated operational chains.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"SeaConditions_operational_chain\">SeaConditions operational chain<\/span><\/h3>\n<p>Concerning the SeaConditions operational chain, the main aim of the DTS is to create the properly processed and formatted data to visualize five days of weather\u2013oceanographic forecasts with three hours of temporal resolution for the first two and a half days and six hours for the others.\n<\/p><p>Dedicated DAS modules, day by day, download the source raw data from the external providers, storing them into the data archive. Raw data are then provided to the SeaConditions operational chains managed by DTS; at the end post-processed outputs are moved to an FTP server. SSA platform procedures are responsible for downloading them as input for the creation of the maps published on the SeaConditions web portal.\n<\/p><p>Dedicated DAS modules, day by day, download the source raw data from the external providers storing them into the data archive. Raw data are then provided to the SeaConditions operational chains managed by DTS; at the end post-processed outputs are moved to an FTP server.\n<\/p><p>This operational chain is based on a workflow manager implemented by a daemonized routine called DTS_SCHEDULER. This daemon manages the parallelization of the data processing tasks using stage-in and stage-out folders as interchange layers with the data archive. The different DTS production phases are represented in Fig. 10, including:\n<\/p>\n<dl><dd>- pre-elaboration phase, which is involved in the data splitting by time and by variables;<\/dd><\/dl>\n<dl><dd>- statistic phase, which aims to compute some statistics as minimum, maximum, mean and standard deviation;<\/dd><\/dl>\n<dl><dd>- interpolation phase, in which the split data resulting from the first phase are filtered by time and some are interpolated in order to increase their spatial resolution; and<\/dd><\/dl>\n<dl><dd>- compression phase, in which the data resulting from the interpolation phase are compressed: Lempel\u2013Ziv deflation, NetCDF4 packing algorithm (from the typical NC FLOAT to NC SHORT reducing the size by about 50 percent) and file zipping with gzip are operations that generally lead to a 70 to 90 percent reduction in file size. If the original NetCDF file occupies 1 GB, at the end of the packing and zipping process it will occupy approximately 100 MB.<\/dd><\/dl>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>One of the main objectives of the TESSA project was to develop a set of operational oceanographic services for the SSA. To reach this goal, advanced technological and software solutions were used as the backbone for the real time dissemination of the oceanographic data to a wide range of users in the maritime, tourism and scientific sectors. In this framework, the relevance of the TESSA data platform is apparent through its capacity to provide efficient and secure data access and strong support to high-level services, such as the DSS.\n<\/p><p>As stated in the introduction, the employment of well-known standards concerning protocols, services and output formats satisfies the FAIR guiding principles for scientific data management and stewardship. Specifically, the designed data platform relies on THREDDS for the publication of the TESSA final products (findable and accessible), while the OPeNDAP service exposes a standard DAP interface (interoperable). In addition, the ISO 19115 standard has been adopted for the definition of a metadata profile containing the mandatory elements of the core metadata set for geographic datasets (interoperable and reusable), and the search and discovery of data are supported by the ESGF data node and by the GeoNetwork information system (findable and accessible). Finally, access to the DSS methods is provided by a machine-readable interface exploiting the JSON format (accessible).\n<\/p><p>In general, the main components of this platform have been designed and developed by taking into account and satisfying a large number of additional requirements such as transparency, robustness, efficiency, fault tolerance and security. Moreover, it is important to emphasize that this platform represents a unique and innovative example of how different components interact with each other to support operational services for the SSA, such as SeaConditions and the other DSS (VISIR, WITOIL, Ocean-SAR, EarlyWarning).\n<\/p><p>Therefore, these features make the TESSA data platform a valid prototype easily adopted to provide an efficient dissemination of maritime data and a consolidation of the management of operational oceanographic activities.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Data_availability\">Data availability<\/span><\/h2>\n<p>Data of subregional models are periodically produced, exploiting the data platform operational chains and published at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/tds.tessa.cmcc.it\" target=\"_blank\">http:\/\/tds.tessa.cmcc.it<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This research was funded by the project TESSA (Technologies for the Situational Sea Awareness; PON01_02823\/2) funded by the Italian Ministry for Education and Research (MIUR).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-DeDominicisMEDSLIK-P113-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeDominicisMEDSLIK-P113_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">De Dominicis, M.; Pinardi, N.; Zodiatis, G.; Lardner, R. (2013). \"MEDSLIK-II, a Lagrangian marine surface oil spill model for short-term forecasting \u2013 Part 1: Theory\". <i>Geoscientific Model Development<\/i> <b>6<\/b> (6): 1851-1869. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fgmd-6-1851-2013\" target=\"_blank\">10.5194\/gmd-6-1851-2013<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MEDSLIK-II%2C+a+Lagrangian+marine+surface+oil+spill+model+for+short-term+forecasting+%E2%80%93+Part+1%3A+Theory&rft.jtitle=Geoscientific+Model+Development&rft.aulast=De+Dominicis%2C+M.%3B+Pinardi%2C+N.%3B+Zodiatis%2C+G.%3B+Lardner%2C+R.&rft.au=De+Dominicis%2C+M.%3B+Pinardi%2C+N.%3B+Zodiatis%2C+G.%3B+Lardner%2C+R.&rft.date=2013&rft.volume=6&rft.issue=6&rft.pages=1851-1869&rft_id=info:doi\/10.5194%2Fgmd-6-1851-2013&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeDominicisMEDSLIK-P213-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeDominicisMEDSLIK-P213_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">De Dominicis, M.; Pinardi, N.; Zodiatis, G.; Archetti, R. (2013). \"MEDSLIK-II, a Lagrangian marine surface oil spill model for short-term forecasting \u2013 Part 2: Numerical simulations and validations\". <i>Geoscientific Model Development<\/i> <b>6<\/b> (6): 1871-1888. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fgmd-6-1871-2013\" target=\"_blank\">10.5194\/gmd-6-1871-2013<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MEDSLIK-II%2C+a+Lagrangian+marine+surface+oil+spill+model+for+short-term+forecasting+%E2%80%93+Part+2%3A+Numerical+simulations+and+validations&rft.jtitle=Geoscientific+Model+Development&rft.aulast=De+Dominicis%2C+M.%3B+Pinardi%2C+N.%3B+Zodiatis%2C+G.%3B+Archetti%2C+R.&rft.au=De+Dominicis%2C+M.%3B+Pinardi%2C+N.%3B+Zodiatis%2C+G.%3B+Archetti%2C+R.&rft.date=2013&rft.volume=6&rft.issue=6&rft.pages=1871-1888&rft_id=info:doi\/10.5194%2Fgmd-6-1871-2013&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoppiniANew16-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CoppiniANew16_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coppini, G.; Jansen, E.; Turrisi, G. et al. (2016). \"A new search-and-rescue service in the Mediterranean Sea: A demonstration of the operational capability and an evaluation of its performance using real case scenarios\". <i>Natural Hazards and Earth System Sciences<\/i> <b>16<\/b> (12): 2713-2727. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fnhess-16-2713-2016\" target=\"_blank\">10.5194\/nhess-16-2713-2016<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+new+search-and-rescue+service+in+the+Mediterranean+Sea%3A+A+demonstration+of+the+operational+capability+and+an+evaluation+of+its+performance+using+real+case+scenarios&rft.jtitle=Natural+Hazards+and+Earth+System+Sciences&rft.aulast=Coppini%2C+G.%3B+Jansen%2C+E.%3B+Turrisi%2C+G.+et+al.&rft.au=Coppini%2C+G.%3B+Jansen%2C+E.%3B+Turrisi%2C+G.+et+al.&rft.date=2016&rft.volume=16&rft.issue=12&rft.pages=2713-2727&rft_id=info:doi\/10.5194%2Fnhess-16-2713-2016&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MannariniVISIR-I16-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MannariniVISIR-I16_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mannarini, G.; Pinardi, N.; Cippini, G. et al. (2016). \"VISIR-I: Small vessels \u2013 Least-time nautical routes using wave forecasts\". <i>Geoscientific Model Development<\/i> <b>9<\/b> (4): 1597-1625. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fgmd-9-1597-2016\" target=\"_blank\">10.5194\/gmd-9-1597-2016<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=VISIR-I%3A+Small+vessels+%E2%80%93+Least-time+nautical+routes+using+wave+forecasts&rft.jtitle=Geoscientific+Model+Development&rft.aulast=Mannarini%2C+G.%3B+Pinardi%2C+N.%3B+Cippini%2C+G.+et+al.&rft.au=Mannarini%2C+G.%3B+Pinardi%2C+N.%3B+Cippini%2C+G.+et+al.&rft.date=2016&rft.volume=9&rft.issue=4&rft.pages=1597-1625&rft_id=info:doi\/10.5194%2Fgmd-9-1597-2016&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MannariniVISIR16-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MannariniVISIR16_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mannarini, G.; Turrisi, G.; D'Anca, A. et al. (2016). \"VISIR: Technological infrastructure of an operational service for safe and efficient navigation in the Mediterranean Sea\". <i>Natural Hazards and Earth System Sciences<\/i> <b>16<\/b> (8): 1791-1806. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fnhess-16-1791-2016\" target=\"_blank\">10.5194\/nhess-16-1791-2016<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=VISIR%3A+Technological+infrastructure+of+an+operational+service+for+safe+and+efficient+navigation+in+the+Mediterranean+Sea&rft.jtitle=Natural+Hazards+and+Earth+System+Sciences&rft.aulast=Mannarini%2C+G.%3B+Turrisi%2C+G.%3B+D%27Anca%2C+A.+et+al.&rft.au=Mannarini%2C+G.%3B+Turrisi%2C+G.%3B+D%27Anca%2C+A.+et+al.&rft.date=2016&rft.volume=16&rft.issue=8&rft.pages=1791-1806&rft_id=info:doi\/10.5194%2Fnhess-16-1791-2016&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LecciSea15-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LecciSea15_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lecci, R.; Coppini, G.; Creti, S. et al. (2015). \"SeaConditions: Present and future sea conditions for safer navigation\". <i>Proceedings from OCEANS 2015 - Genova<\/i> <b>2015<\/b>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FOCEANS-Genova.2015.7271764\" target=\"_blank\">10.1109\/OCEANS-Genova.2015.7271764<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SeaConditions%3A+Present+and+future+sea+conditions+for+safer+navigation&rft.jtitle=Proceedings+from+OCEANS+2015+-+Genova&rft.aulast=Lecci%2C+R.%3B+Coppini%2C+G.%3B+Creti%2C+S.+et+al.&rft.au=Lecci%2C+R.%3B+Coppini%2C+G.%3B+Creti%2C+S.+et+al.&rft.date=2015&rft.volume=2015&rft_id=info:doi\/10.1109%2FOCEANS-Genova.2015.7271764&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilkinsonTheFAIR16-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilkinsonTheFAIR16_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175\" target=\"_blank\">\"The FAIR Guiding Principles for scientific data management and stewardship\"<\/a>. <i>Scientific Data<\/i> <b>3<\/b>: 160018. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fsdata.2016.18\" target=\"_blank\">10.1038\/sdata.2016.18<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4792175\/\" target=\"_blank\">PMC4792175<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26978244\" target=\"_blank\">26978244<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+FAIR+Guiding+Principles+for+scientific+data+management+and+stewardship&rft.jtitle=Scientific+Data&rft.aulast=Wilkinson%2C+M.D.%3B+Dumontier%2C+M.%3B+Aalbersberg%2C+I.J.+et+al.&rft.au=Wilkinson%2C+M.D.%3B+Dumontier%2C+M.%3B+Aalbersberg%2C+I.J.+et+al.&rft.date=2016&rft.volume=3&rft.pages=160018&rft_id=info:doi\/10.1038%2Fsdata.2016.18&rft_id=info:pmc\/PMC4792175&rft_id=info:pmid\/26978244&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4792175&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OGCnetCDF-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OGCnetCDF_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.opengeospatial.org\/standards\/netcdf\" target=\"_blank\">\"OGC network Common Data Form (netCDF) standards suite\"<\/a>. Open Geospatial Consortium<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.opengeospatial.org\/standards\/netcdf\" target=\"_blank\">http:\/\/www.opengeospatial.org\/standards\/netcdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OGC+network+Common+Data+Form+%28netCDF%29+standards+suite&rft.atitle=&rft.pub=Open+Geospatial+Consortium&rft_id=http%3A%2F%2Fwww.opengeospatial.org%2Fstandards%2Fnetcdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UnidataNetCDF-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UnidataNetCDF_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.unidata.ucar.edu\/software\/netcdf\/\" target=\"_blank\">\"Network Common Data Form (NetCDF)\"<\/a>. <i>Unidata<\/i>. University Corporation for Atmospheric Research<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.unidata.ucar.edu\/software\/netcdf\/\" target=\"_blank\">https:\/\/www.unidata.ucar.edu\/software\/netcdf\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Network+Common+Data+Form+%28NetCDF%29&rft.atitle=Unidata&rft.pub=University+Corporation+for+Atmospheric+Research&rft_id=https%3A%2F%2Fwww.unidata.ucar.edu%2Fsoftware%2Fnetcdf%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CopernicusMEMS-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CopernicusMEMS_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/marine.copernicus.eu\/\" target=\"_blank\">\"Copernicus Marine Environment Monitoring Service\"<\/a>. European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/marine.copernicus.eu\/\" target=\"_blank\">http:\/\/marine.copernicus.eu\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Copernicus+Marine+Environment+Monitoring+Service&rft.atitle=&rft.pub=European+Commission&rft_id=http%3A%2F%2Fmarine.copernicus.eu%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BahurelOcean09-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BahurelOcean09_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bahurel, P.; Adragna, F.; Bell, M.J. et al. (2009). \"Ocean Monitoring and Forecasting Core Services, the European MyOcean Example\". <i>Proceedings of OceanObs'09: Sustained Ocean Observations and Information for Society<\/i> <b>2009<\/b>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5270%2FOceanObs09.pp.02\" target=\"_blank\">10.5270\/OceanObs09.pp.02<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ocean+Monitoring+and+Forecasting+Core+Services%2C+the+European+MyOcean+Example&rft.jtitle=Proceedings+of+OceanObs%2709%3A+Sustained+Ocean+Observations+and+Information+for+Society&rft.aulast=Bahurel%2C+P.%3B+Adragna%2C+F.%3B+Bell%2C+M.J.+et+al.&rft.au=Bahurel%2C+P.%3B+Adragna%2C+F.%3B+Bell%2C+M.J.+et+al.&rft.date=2009&rft.volume=2009&rft_id=info:doi\/10.5270%2FOceanObs09.pp.02&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Copernicus-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Copernicus_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.copernicus.eu\/\" target=\"_blank\">\"Copernicus\"<\/a>. European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.copernicus.eu\/\" target=\"_blank\">http:\/\/www.copernicus.eu\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Copernicus&rft.atitle=&rft.pub=European+Commission&rft_id=http%3A%2F%2Fwww.copernicus.eu%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO19115:2003-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISO19115:2003_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">ISO\/TC 211 Geographic information\/Geomatics (May 2003). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/26020.html\" target=\"_blank\">\"ISO 19115:2003 Geographic information -- Metadata\"<\/a>. International Organization for Standardization<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/26020.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/26020.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO+19115%3A2003+Geographic+information+--+Metadata&rft.atitle=&rft.aulast=ISO%2FTC+211+Geographic+information%2FGeomatics&rft.au=ISO%2FTC+211+Geographic+information%2FGeomatics&rft.date=May+2003&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F26020.html&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO19139:2007-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISO19139:2007_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">ISO\/TC 211 Geographic information\/Geomatics (April 2007). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/32557.html\" target=\"_blank\">\"ISO\/TS 19139:2007 Geographic information -- Metadata -- XML schema implementation\"<\/a>. International Organization for Standardization<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/32557.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/32557.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO%2FTS+19139%3A2007+Geographic+information+--+Metadata+--+XML+schema+implementation&rft.atitle=&rft.aulast=ISO%2FTC+211+Geographic+information%2FGeomatics&rft.au=ISO%2FTC+211+Geographic+information%2FGeomatics&rft.date=April+2007&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F32557.html&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-INSPIRE-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-INSPIRE_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/inspire.ec.europa.eu\/\" target=\"_blank\">\"INSPIRE: Infrastructure for Spatial Information in Europe\"<\/a>. European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/inspire.ec.europa.eu\/\" target=\"_blank\">http:\/\/inspire.ec.europa.eu\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=INSPIRE%3A+Infrastructure+for+Spatial+Information+in+Europe&rft.atitle=&rft.pub=European+Commission&rft_id=http%3A%2F%2Finspire.ec.europa.eu%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EuropeanDirective07-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EuropeanDirective07_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">European Parliament (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/data.europa.eu\/eli\/dir\/2007\/2\/oj\" target=\"_blank\">\"Directive 2007\/2\/EC of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in the European Community (INSPIRE)\"<\/a>. <i>Official Journal of the European Union<\/i> <b>L108<\/b> (25.4.2007)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/data.europa.eu\/eli\/dir\/2007\/2\/oj\" target=\"_blank\">http:\/\/data.europa.eu\/eli\/dir\/2007\/2\/oj<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Directive+2007%2F2%2FEC+of+the+European+Parliament+and+of+the+Council+of+14+March+2007+establishing+an+Infrastructure+for+Spatial+Information+in+the+European+Community+%28INSPIRE%29&rft.jtitle=Official+Journal+of+the+European+Union&rft.aulast=European+Parliament&rft.au=European+Parliament&rft.date=2007&rft.volume=L108&rft.issue=25.4.2007&rft_id=http%3A%2F%2Fdata.europa.eu%2Feli%2Fdir%2F2007%2F2%2Foj&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CinquiniTheEarth14-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CinquiniTheEarth14_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cinquini, L.; Crichton, D.; Mattmann, C. et al. (2014). \"The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data\". <i>Future Generation Computer Systems<\/i> <b>36<\/b> (July 2014): 400\u2013417. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.future.2013.07.002\" target=\"_blank\">10.1016\/j.future.2013.07.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Earth+System+Grid+Federation%3A+An+open+infrastructure+for+access+to+distributed+geospatial+data&rft.jtitle=Future+Generation+Computer+Systems&rft.aulast=Cinquini%2C+L.%3B+Crichton%2C+D.%3B+Mattmann%2C+C.+et+al.&rft.au=Cinquini%2C+L.%3B+Crichton%2C+D.%3B+Mattmann%2C+C.+et+al.&rft.date=2014&rft.volume=36&rft.issue=July+2014&rft.pages=400%E2%80%93417&rft_id=info:doi\/10.1016%2Fj.future.2013.07.002&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ESGF-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ESGF_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/esgf.llnl.gov\/\" target=\"_blank\">\"Earth System Grid Federation\"<\/a>. ESGF Executive Committee<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/esgf.llnl.gov\/\" target=\"_blank\">https:\/\/esgf.llnl.gov\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Earth+System+Grid+Federation&rft.atitle=&rft.pub=ESGF+Executive+Committee&rft_id=https%3A%2F%2Fesgf.llnl.gov%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EMODnet-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EMODnet_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.emodnet-mediterranean.eu\/\" target=\"_blank\">\"EMODnet: European Marine Observation and Data Network\"<\/a>. European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.emodnet-mediterranean.eu\/\" target=\"_blank\">http:\/\/www.emodnet-mediterranean.eu\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=EMODnet%3A+European+Marine+Observation+and+Data+Network&rft.atitle=&rft.pub=European+Commission&rft_id=http%3A%2F%2Fwww.emodnet-mediterranean.eu%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoussatEMODnet16-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoussatEMODnet16_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moussat, E.; Pinardi, N.; Manzella, G. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/meetingorganizer.copernicus.org\/EGU2016\/EGU2016-9103.pdf\" target=\"_blank\">\"EMODnet MedSea Checkpoint for sustainable Blue Growth\"<\/a>. <i>Geophysical Research Abstracts<\/i> <b>18<\/b>: 9103<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/meetingorganizer.copernicus.org\/EGU2016\/EGU2016-9103.pdf\" target=\"_blank\">http:\/\/meetingorganizer.copernicus.org\/EGU2016\/EGU2016-9103.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EMODnet+MedSea+Checkpoint+for+sustainable+Blue+Growth&rft.jtitle=Geophysical+Research+Abstracts&rft.aulast=Moussat%2C+E.%3B+Pinardi%2C+N.%3B+Manzella%2C+G.+et+al.&rft.au=Moussat%2C+E.%3B+Pinardi%2C+N.%3B+Manzella%2C+G.+et+al.&rft.date=2016&rft.volume=18&rft.pages=9103&rft_id=http%3A%2F%2Fmeetingorganizer.copernicus.org%2FEGU2016%2FEGU2016-9103.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SeaDataNet-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SeaDataNet_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.seadatanet.org\/\" target=\"_blank\">\"SeaDataNet: Pan-European Infrastructure for Ocean & Marine Data Management\"<\/a>. European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.seadatanet.org\/\" target=\"_blank\">https:\/\/www.seadatanet.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SeaDataNet%3A+Pan-European+Infrastructure+for+Ocean+%26+Marine+Data+Management&rft.atitle=&rft.pub=European+Commission&rft_id=https%3A%2F%2Fwww.seadatanet.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CLIPC-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CLIPC_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.clipc.eu\/home\" target=\"_blank\">\"CLIPC: Climate Information Portal\"<\/a>. European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.clipc.eu\/home\" target=\"_blank\">http:\/\/www.clipc.eu\/home<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CLIPC%3A+Climate+Information+Portal&rft.atitle=&rft.pub=European+Commission&rft_id=http%3A%2F%2Fwww.clipc.eu%2Fhome&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISOTC2011StandardsGuide-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISOTC2011StandardsGuide_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.isotc211.org\/Outreach\/ISO_TC_211_Standards_Guide.pdf\" target=\"_blank\">\"Standards Guide: ISO\/TC 211 Geographic information\/Geomatics\"<\/a> (PDF). International Organization for Standardization. 01 June 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.isotc211.org\/Outreach\/ISO_TC_211_Standards_Guide.pdf\" target=\"_blank\">http:\/\/www.isotc211.org\/Outreach\/ISO_TC_211_Standards_Guide.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Standards+Guide%3A+ISO%2FTC+211+Geographic+information%2FGeomatics&rft.atitle=&rft.date=01+June+2009&rft.pub=International+Organization+for+Standardization&rft_id=http%3A%2F%2Fwww.isotc211.org%2FOutreach%2FISO_TC_211_Standards_Guide.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScalasTESSA17-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScalasTESSA17_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scalas, M.; Palmalisa, M.; Tedesco, L. et al. (2017). \"TESSA: Design and implementation of a platform for situational sea awareness\". <i>Natural Hazards and Earth System Sciences<\/i> <b>17<\/b> (2): 185-196. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fnhess-17-185-2017\" target=\"_blank\">10.5194\/nhess-17-185-2017<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=TESSA%3A+Design+and+implementation+of+a+platform+for+situational+sea+awareness&rft.jtitle=Natural+Hazards+and+Earth+System+Sciences&rft.aulast=Scalas%2C+M.%3B+Palmalisa%2C+M.%3B+Tedesco%2C+L.+et+al.&rft.au=Scalas%2C+M.%3B+Palmalisa%2C+M.%3B+Tedesco%2C+L.+et+al.&rft.date=2017&rft.volume=17&rft.issue=2&rft.pages=185-196&rft_id=info:doi\/10.5194%2Fnhess-17-185-2017&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-THREDDS-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-THREDDS_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.unidata.ucar.edu\/software\/thredds\/current\/tds\/\" target=\"_blank\">\"THREDDS Data Server (TDS)\"<\/a>. <i>Unidata<\/i>. University Corporation for Atmospheric Research<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.unidata.ucar.edu\/software\/thredds\/current\/tds\/\" target=\"_blank\">http:\/\/www.unidata.ucar.edu\/software\/thredds\/current\/tds\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=THREDDS+Data+Server+%28TDS%29&rft.atitle=Unidata&rft.pub=University+Corporation+for+Atmospheric+Research&rft_id=http%3A%2F%2Fwww.unidata.ucar.edu%2Fsoftware%2Fthredds%2Fcurrent%2Ftds%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuarnieriTheAdri08-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuarnieriTheAdri08_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Guarnieri, A.; Oddo, P.; Pastore, M.; Pinardi, N. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.584.9289&rep=rep1&type=pdf\" target=\"_blank\">\"The Adriatic Basin Forecasting System: New model and system development\"<\/a>. <i>Proceeding of 5th EuroGOOS Conference, Exeter<\/i> <b>2008<\/b><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.584.9289&rep=rep1&type=pdf\" target=\"_blank\">http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.584.9289&rep=rep1&type=pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Adriatic+Basin+Forecasting+System%3A+New+model+and+system+development&rft.jtitle=Proceeding+of+5th+EuroGOOS+Conference%2C+Exeter&rft.aulast=Guarnieri%2C+A.%3B+Oddo%2C+P.%3B+Pastore%2C+M.%3B+Pinardi%2C+N.&rft.au=Guarnieri%2C+A.%3B+Oddo%2C+P.%3B+Pastore%2C+M.%3B+Pinardi%2C+N.&rft.date=2008&rft.volume=2008&rft_id=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.584.9289%26rep%3Drep1%26type%3Dpdf&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OddoANumer05-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OddoANumer05_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Oddo, P.; Pinardi, N.; Zavatarelli, M. (2005). \"A numerical study of the interannual variability of the Adriatic Sea (2000\u20132002)\". <i>Science of The Total Environment<\/i> <b>353<\/b> (1\u20133): 39\u201356. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.scitotenv.2005.09.061\" target=\"_blank\">10.1016\/j.scitotenv.2005.09.061<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16257438\" target=\"_blank\">16257438<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+numerical+study+of+the+interannual+variability+of+the+Adriatic+Sea+%282000%E2%80%932002%29&rft.jtitle=Science+of+The+Total+Environment&rft.aulast=Oddo%2C+P.%3B+Pinardi%2C+N.%3B+Zavatarelli%2C+M.&rft.au=Oddo%2C+P.%3B+Pinardi%2C+N.%3B+Zavatarelli%2C+M.&rft.date=2005&rft.volume=353&rft.issue=1%E2%80%933&rft.pages=39%E2%80%9356&rft_id=info:doi\/10.1016%2Fj.scitotenv.2005.09.061&rft_id=info:pmid\/16257438&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OddoTheAdri06-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OddoTheAdri06_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Oddo, P.; Pinardi, N.; Zavatarelli, M.; Coluccelli, A. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/hrcak.srce.hr\/8550\" target=\"_blank\">\"The Adriatic Basin forecasting system\"<\/a>. <i>Acta Adriatica<\/i> <b>47<\/b> (Sup. 1): 169\u2013184<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/hrcak.srce.hr\/8550\" target=\"_blank\">http:\/\/hrcak.srce.hr\/8550<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Adriatic+Basin+forecasting+system&rft.jtitle=Acta+Adriatica&rft.aulast=Oddo%2C+P.%3B+Pinardi%2C+N.%3B+Zavatarelli%2C+M.%3B+Coluccelli%2C+A.&rft.au=Oddo%2C+P.%3B+Pinardi%2C+N.%3B+Zavatarelli%2C+M.%3B+Coluccelli%2C+A.&rft.date=2006&rft.volume=47&rft.issue=Sup.+1&rft.pages=169%E2%80%93184&rft_id=http%3A%2F%2Fhrcak.srce.hr%2F8550&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MadecNEMOOcean08-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MadecNEMOOcean08_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Madec, G. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.nemo-ocean.eu\/About-NEMO\/Reference-manuals\" target=\"_blank\">\"NEMO Ocean General Circulation Model Reference Manual, Internal Report\"<\/a>. LODYC\/IPSL<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.nemo-ocean.eu\/About-NEMO\/Reference-manuals\" target=\"_blank\">http:\/\/www.nemo-ocean.eu\/About-NEMO\/Reference-manuals<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NEMO+Ocean+General+Circulation+Model+Reference+Manual%2C+Internal+Report&rft.atitle=&rft.aulast=Madec%2C+G.&rft.au=Madec%2C+G.&rft.date=2008&rft.pub=LODYC%2FIPSL&rft_id=http%3A%2F%2Fwww.nemo-ocean.eu%2FAbout-NEMO%2FReference-manuals&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OddoANested09-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OddoANested09_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Oddo, P.; Adani, M.; Pinardi, N. et al. (2009). \"A nested Atlantic-Mediterranean Sea general circulation model for operational forecasting\". <i>Ocean Science<\/i> <b>5<\/b> (4): 461-473. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fos-5-461-2009\" target=\"_blank\">10.5194\/os-5-461-2009<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+nested+Atlantic-Mediterranean+Sea+general+circulation+model+for+operational+forecasting&rft.jtitle=Ocean+Science&rft.aulast=Oddo%2C+P.%3B+Adani%2C+M.%3B+Pinardi%2C+N.+et+al.&rft.au=Oddo%2C+P.%3B+Adani%2C+M.%3B+Pinardi%2C+N.+et+al.&rft.date=2009&rft.volume=5&rft.issue=4&rft.pages=461-473&rft_id=info:doi\/10.5194%2Fos-5-461-2009&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FedericoCoastal17-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FedericoCoastal17_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Federico, I.; Pinardi, N.; Coppini, G. et al. (2017). \"Coastal ocean forecasting with an unstructured grid model in the southern Adriatic and northern Ionian seas\". <i>Natural Hazards and Earth System Sciences<\/i> <b>17<\/b> (1): 45\u201359. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fnhess-17-45-2017\" target=\"_blank\">10.5194\/nhess-17-45-2017<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Coastal+ocean+forecasting+with+an+unstructured+grid+model+in+the+southern+Adriatic+and+northern+Ionian+seas&rft.jtitle=Natural+Hazards+and+Earth+System+Sciences&rft.aulast=Federico%2C+I.%3B+Pinardi%2C+N.%3B+Coppini%2C+G.+et+al.&rft.au=Federico%2C+I.%3B+Pinardi%2C+N.%3B+Coppini%2C+G.+et+al.&rft.date=2017&rft.volume=17&rft.issue=1&rft.pages=45%E2%80%9359&rft_id=info:doi\/10.5194%2Fnhess-17-45-2017&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OGCOpenGIS06-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OGCOpenGIS06_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">de la Beaujardiere, J., ed. (15 March 2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/portal.opengeospatial.org\/files\/?artifact_id=14416\" target=\"_blank\">\"OpenGIS Web Map Server Implementation Specification\"<\/a>. Open Geospatial Consortium, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/portal.opengeospatial.org\/files\/?artifact_id=14416\" target=\"_blank\">http:\/\/portal.opengeospatial.org\/files\/?artifact_id=14416<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 February 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OpenGIS+Web+Map+Server+Implementation+Specification&rft.atitle=&rft.date=15+March+2006&rft.pub=Open+Geospatial+Consortium%2C+Inc&rft_id=http%3A%2F%2Fportal.opengeospatial.org%2Ffiles%2F%3Fartifact_id%3D14416&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OPeNDAP-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OPeNDAP_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.opendap.org\/about\" target=\"_blank\">\"About OPeNDAP\"<\/a>. OPeNDAP<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.opendap.org\/about\" target=\"_blank\">https:\/\/www.opendap.org\/about<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=About+OPeNDAP&rft.atitle=&rft.pub=OPeNDAP&rft_id=https%3A%2F%2Fwww.opendap.org%2Fabout&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchmuckGPFS02-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchmuckGPFS02_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1083323.1083349\" target=\"_blank\">\"GPFS: A Shared-Disk File System for Large Computing Clusters\"<\/a>. <i>Proceedings of the 1st USENIX Conference on File and Storage Technologies<\/i> <b>2002<\/b>: 19. 2002<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1083323.1083349\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?id=1083323.1083349<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GPFS%3A+A+Shared-Disk+File+System+for+Large+Computing+Clusters&rft.jtitle=Proceedings+of+the+1st+USENIX+Conference+on+File+and+Storage+Technologies&rft.date=2002&rft.volume=2002&rft.pages=19&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fid%3D1083323.1083349&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IHO-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IHO_35-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.iho.int\/srv1\" target=\"_blank\">\"International Hydrographic Organization\"<\/a>. International Hydrographic Organization<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.iho.int\/srv1\" target=\"_blank\">http:\/\/www.iho.int\/srv1<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=International+Hydrographic+Organization&rft.atitle=&rft.pub=International+Hydrographic+Organization&rft_id=http%3A%2F%2Fwww.iho.int%2Fsrv1&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeoNetworkOS-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeoNetworkOS_36-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/geonetwork-opensource.org\/\" target=\"_blank\">\"GeoNetwork opensource\"<\/a>. Open Source Geospatial Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/geonetwork-opensource.org\/\" target=\"_blank\">http:\/\/geonetwork-opensource.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GeoNetwork+opensource&rft.atitle=&rft.pub=Open+Source+Geospatial+Foundation&rft_id=http%3A%2F%2Fgeonetwork-opensource.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SeaConditions-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SeaConditions_37-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.sea-conditions.com\/en\/\" target=\"_blank\">\"SeaConditions\"<\/a>. Links Management and Technology S.p.a<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.sea-conditions.com\/en\/\" target=\"_blank\">http:\/\/www.sea-conditions.com\/en\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 January 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SeaConditions&rft.atitle=&rft.pub=Links+Management+and+Technology+S.p.a&rft_id=http%3A%2F%2Fwww.sea-conditions.com%2Fen%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. References in the original work were alphabetical; this website organizes references in order of appearance, by design. Grammar was cleaned up in several places. A URL to Unidata's NetCDF page was originally incorrect and updated for this version. URLs were added to the ISO and other standards references for improved reference information.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181459\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.774 seconds\nReal time usage: 0.815 seconds\nPreprocessor visited node count: 25464\/1000000\nPreprocessor generated node count: 34283\/1000000\nPost\u2010expand include size: 165834\/2097152 bytes\nTemplate argument size: 58281\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 755.764 1 - -total\n 86.69% 655.148 1 - Template:Reflist\n 74.42% 562.406 37 - Template:Citation\/core\n 43.35% 327.648 18 - Template:Cite_journal\n 35.42% 267.704 19 - Template:Cite_web\n 8.09% 61.145 1 - Template:Infobox_journal_article\n 7.78% 58.821 1 - Template:Infobox\n 4.65% 35.120 80 - Template:Infobox\/row\n 3.82% 28.893 39 - Template:Citation\/make_link\n 3.53% 26.648 16 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10089-0!*!0!!en!5!* and timestamp 20181214181458 and revision id 30339\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products\">https:\/\/www.limswiki.org\/index.php\/Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","e86f9fcd9da316970f965a883c42e462_images":["https:\/\/www.limswiki.org\/images\/e\/e2\/Fig1_D%27Anca_NatHazEarth2017_17-2.png","https:\/\/www.limswiki.org\/images\/e\/e5\/Fig2_D%27Anca_NatHazEarth2017_17-2.png","https:\/\/www.limswiki.org\/images\/4\/43\/Fig3_D%27Anca_NatHazEarth2017_17-2.png","https:\/\/www.limswiki.org\/images\/8\/8e\/Fig4_D%27Anca_NatHazEarth2017_17-2.png","https:\/\/www.limswiki.org\/images\/2\/29\/Fig5_D%27Anca_NatHazEarth2017_17-2.png","https:\/\/www.limswiki.org\/images\/1\/11\/Fig6_D%27Anca_NatHazEarth2017_17-2.png","https:\/\/www.limswiki.org\/images\/a\/a8\/Fig7_D%27Anca_NatHazEarth2017_17-2.png","https:\/\/www.limswiki.org\/images\/e\/e5\/Fig8_D%27Anca_NatHazEarth2017_17-2.png","https:\/\/www.limswiki.org\/images\/1\/15\/Fig9_D%27Anca_NatHazEarth2017_17-2.png"],"e86f9fcd9da316970f965a883c42e462_timestamp":1544811298,"021382bc6a00c9f9a06aef062ae61eaa_type":"article","021382bc6a00c9f9a06aef062ae61eaa_title":"Selecting a laboratory information management system for biorepositories in low- and middle-income countries: The H3Africa experience and lessons learned (Kyobe et al. 2017)","021382bc6a00c9f9a06aef062ae61eaa_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned","021382bc6a00c9f9a06aef062ae61eaa_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Selecting a laboratory information management system for biorepositories in low- and middle-income countries: The H3Africa experience and lessons learned\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nSelecting a laboratory information management system for biorepositories in low- and middle-income countries: The H3Africa experience and lessons learnedJournal\n \nBiopreservation and BiobankingAuthor(s)\n \n\n List of authors\n Kyobe, S.; Musinguzi, H.; Lwanga, N.; Kezimbira, D.; Kigozi, E.; Katabazi, F.A.; Wayengera, M.; Joloba, M.L.; Abayomi, E.A.; Swanepoel, C.; Abimiku, A.; Croxton, T.; Ozumba, P.; Thankgod, A.; Christoffels, A.; van Zyl, L.; Mayne, E.S.; Kader, M.; Swartz, G.; H3Africa Biorepository PI Working Group\nAuthor affiliation(s)\n \n\n List of affiliations\n Makerere University College of Health Sciences, Tygerberg Hospital, Stellenbosch University, University of Maryland - College Park, Institute of Human Virology - Nigeria, South African National Bioinformatics Institute, University of the Western Cape, Clinical Laboratory Services - Wits Health Consortium\nPrimary contact\n \nEmail: samkyobe at chs dot mak dot ac dot ugYear published\n \n2017Volume and issue\n \n15 (2)Page(s)\n \n111-115DOI\n \n10.1089\/bio.2017.0006ISSN\n \n1947-5543Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/online.liebertpub.com\/doi\/full\/10.1089\/bio.2017.0006Download\n \nhttp:\/\/online.liebertpub.com\/doi\/pdfplus\/10.1089\/bio.2017.0006 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Materials and methods \n4 Results and discussion \n\n4.1 LIMS selection at the H3Africa biorepositories \n4.2 Checklist for choosing a LIMS \n4.3 Commercial versus open-source LIMS \n4.4 Financing of LIMS in the LMIC biorepositories \n4.5 IT infrastructure \n4.6 User support services \n4.7 LIMS harmonization in the H3Africa biorepositories \n4.8 LIMS sustainability \n\n\n5 Conclusion \n6 Acknowledgements \n7 Author disclosure statement \n8 References \n9 Notes \n\n\n\nAbstract \nBiorepositories in Africa need significant infrastructural support to meet International Society for Biological and Environmental Repositories (ISBER) Best Practices to support population-based genomics research. ISBER recommends a biorepository information management system which can manage workflows from biospecimen receipt to distribution. The H3Africa Initiative set out to develop regional African biorepositories where Uganda, Nigeria, and South Africa were successfully awarded grants to develop the state-of-the-art biorepositories. The biorepositories carried out an elaborate process to evaluate and choose a laboratory information management system (LIMS) with the aim of integrating the three geographically distinct sites. In this article, we review the processes, African experience, and lessons learned, and we make recommendations for choosing a biorepository LIMS in the African context.\n\nIntroduction \nA biorepository requires the linkage of high-quality material to data housed in a laboratory information management system (LIMS), which tracks each sample. A founding principle of H3Africa is to make sure that DNA (and possibly other biological material) would be properly stored in biorepositories for future research purposes.[1][2] The value of this material is partly determined by the associated phenotypic data. Management of this data and sample tracking in compliance with national and international best practices and ethical guidelines requires a refined data management system.[3][4] According to the International Society for Biological and Environmental Repositories (ISBER) Best Practices, a computer-based inventory system is necessary to track the location and annotation of every specimen in the biorepository.[3] The system should also track significant events during a sample's existence from collection to destruction, including sample thaws, receipt and\/or processing delays, processing, transfer of the sample within the repository, specimen distribution and return, and destruction.[3][4][5] These ISBER Best Practices, as well as others from around the world, are under consideration to codify International Organization for Standardization norms as the basis for a new international accreditation program for biorepositories.[6] There are several factors that hinder successful implementation of a biorepository LIMS in Africa. This article discusses the H3Africa Biorepositories experiences and provides recommendations for the process of evaluating, choosing, and deployment of a sustainable biorepository LIMS in low- and middle-income countries (LMIC).\n\nMaterials and methods \nIn 2012 and 2013, the National Institutes of Health-funded H3Africa Program established three regional biorepositories in Africa: Uganda, South Africa, and Nigeria. The biorepositories were developed to archive and facilitate international scientific access to well-curated biospecimens collected by research projects in the H3Africa Consortium Program. To facilitate international access to biospecimens, it was desirable that the biorepositories set up an interoperable LIMS for data integration and exchange. At the time of funding, each of the three biorepositories had a LIMS for their current capacity. The biorepositories adopted a user requirements checklist provided by Autoscribe Informatics (Berkshire, UK) for the LIMS evaluation. The checklist consists of 15 key user requirements, including, but not limited to, general system information, system configuration and customization, user access and security, functionality, static data tables, reporting system, and system support. Six commercial LIMS companies were invited to respond to the questions and requirements in the evaluation tool. Two open-source LIMS were self-evaluated by the H3Africa biorepositories in Uganda and South Africa. The returned results from three commercial LIMS vendors were combined for analysis and discussed during conference calls and online correspondence. Two biorepository consultants provided advice during the evaluation process. The technical description of how the commercial and open-source LIMS compared with each other is beyond the scope of this article. However, we describe the experience and lessons drawn from the LIMS evaluation process from an African\/LIMC perspective.\n\nResults and discussion \nLIMS selection at the H3Africa biorepositories \nFollowing the evaluation process, the H3Africa biorepositories made decisions on their biorepository LIMS. The Institute of Human Virology Nigerian\u2013H3Africa Biorepository (I-HAB) in Nigeria decided to upgrade the current LIMS with the latest software updates from the vendor. The Integrated Biorepository of H3Africa Uganda (IBRH3AU) in Uganda decided to acquire a new LIMS and migrate the data from the old platform to the new LIMS. The Clinical Laboratory Services\u2013H3Africa Biorepository (CLS) in South Africa decided to acquire a new LIMS platform in addition to the existing LIMS. Both CLS and IBRH3AU chose the same new LIMS platforms. I-HAB was convinced that an upgrade was necessary and sufficient to achieve the interoperable requirements of the three program biorepositories.\n\nChecklist for choosing a LIMS \nThe H3Africa project specifications required that the three biorepositories' workflows would be harmonized through interoperable LIMS to enable data integration and exchange. Following the evaluation process, the following key elements were considered while choosing a LIMS for the project: (1) customizability and usability; (2) interoperability with other LIMS; (3) access to revisions, updates, patches, and maintenance releases; (4) cost and access to technical support services; (5) maintenance and associated costs; (6) multi-user\/site support; (7) robustness to handle large volumes of sample information; (8) security systems (audit trail, user roles, and privileges, etc.); and (9) type of (open-source or commercial) LIMS. Table 1 represents a summarized checklist of the elements we considered to be key while choosing a biorepository LIMS. A detailed checklist can be accessed via biorepository.h3africa.org website.\n\n\n\n\n\n\n\nTable 1. Summarized checklist to consider while choosing a biorepository laboratory information management system. A detailed checklist can be accessed via biorepository.h3africa.org. CFR = Code of Federal Regulations; LIMS = laboratory information management system\n\n\nCategory\n\nConsiderations\n\n\nLIMS general information\n\n\u2022\u2002What development tools have been used in the creation of the LIMS (e.g., Microsoft C#)?\r\n\u2022\u2002Is the system scalable and therefore suitable for small organizations as well as global corporations?\r\n\u2022\u2002What are the system requirements?\r\n\u2022\u2002Does the system allow concurrent operation of three or more database groups in the standard system?\n\n\nSystem configuration\/customization\n\n\u2022\u2002Is your system configurable so that it can exactly meet our requirements? If yes, what are the requirements? If no, what are the alternatives?\r\n\u2022\u2002Is the system configurable by the nonprogrammer and without use of special languages?\r\n\u2022\u2002Are user configuration changes supported by the vendor? Is there any extra cost?\n\n\nUser access\/security\n\n\u2022\u2002What is the security access system in detail?\r\n\u2022\u2002Does your system password requirements comply with 21CFR Part 11?\n\n\nFunctionality\u2014sample\/work registration\n\n\u2022\u2002Does your system support the following types of registration: Single sample; Single sample with copy feature; Batch registration with\/without copy feature; Bar-code support; Registration templates; Registration from external system and\/or scheduler; Spreadsheet style registration, including data capture from Excel; Can details of submitters and sample types be viewed from the registration screen; Can reports (e.g., worksheets and labels be automatically generated)?\r\n\u2022\u2002Does system allow fields to be populated automatically by defining a default value within the system which can be overtyped by a user with suitable authority?\n\n\nFunctionality\u2014sample receipt\n\n\u2022\u2002Is a sample receipt function provided with the system? Useful when some work is preregistered before the availability of the samples. The receipt function is needed to track the arrival of the samples. Useful in checking whether all expected samples arrive in the laboratory and the time interval between registration and receipt for each sample can be measured. The sample turnaround time in the laboratory should be measured from the receipt date and time.\r\n\u2022\u2002Does sample receipt allow for single sample, multiple sample, batch sample, and global sample receipt utilizing bar-codes where needed?\n\n\nFunctionality\u2014sample preparation\n\n\u2022\u2002Is a sample preparation function provided with the system? This would be used to indicate that samples must complete the preparation stage before they are ready for testing.\r\n\u2022\u2002Does sample preparation allow for single sample, multiple sample, batch sample, and global sample preparation?\n\n\nFunctionality\u2014result entry\n\n\u2022\u2002Does system include: Result entry by sample\u2014entry of any\/all test results for a single sample; Result entry by test\u2014entry of one test result for multiple samples; Result entry for multiple samples and multiple tests in spreadsheet style; Result import from a variety of sources including files and instruments; Viewing of results previously entered using same selection criteria as for selection of samples for result entry; Viewing of test status.\n\n\nReporting\n\n\u2022\u2002What reporting tools does the system support?\r\n\u2022\u2002Is event triggered reporting\u2014reports generated by sample status change, for example\u2013included?\n\n\nSystem support\n\n\u2022\u2002If I have a support question can I telephone a help-desk and immediately talk to a technical person familiar with my system? If yes, where would this person be located and what time can I call them? If no, what support scheme is in place?\r\n\u2022\u2002Does the vendor have global coverage if relevant? If so what are the support center locations?\r\n\u2022\u2002Are new version upgrades supplied to customers at no cost?\r\n\u2022\u2002How much effort is typically needed for the implementation of an upgrade and what do I have to pay?\r\n\u2022\u2002Will an upgrade preserve my configuration or custom code as well as my data?\r\n\u2022\u2002Do you guarantee that we will always be able to upgrade to the next version?\n\n\nMiscellaneous\n\n\u2022\u2002Does system allow storage of BLOB files, for example, pictures and documents (Consent forms) associated with a sample or a test?\r\n\u2022\u2002Can a document management capability be fully integrated within the system?\r\n\u2022\u2002Describe the sample tracking features of your LIMS. How is this used to monitor inventory for example?\r\n\u2022\u2002Can I have separate databases for different departments within my organization (they may be in different locations)? Can these databases be configured differently?\n\n\n\nCommercial versus open-source LIMS \nCommercial LIMS are systems whose source codes are developed for sale and need authorization from vendors for licensed use. Open-source LIMS are systems whose source codes are available for distribution at no cost.[7] During the H3Africa biorepository LIMS evaluation phase, it was decided to implement a commercial LIMS in support of the H3Africa biospecimen collections. Commercial LIMS are much more expensive upfront than open-source LIMS, are less flexible for end-user adaptations, but do not need local expertise for support. Despite some features of modified open-source LIMS that might seem more applicable to the African setting, there were significant concerns about the stability of such systems and the lack of standardization. In addition, it was clear that the adaptation and maintenance of such a system would need highly specialized staff at three biorepositories and that this approach could create differences among the biorepositories, which could cause potential delays in interbiorepository transfer of data and material.\n\nFinancing of LIMS in the LMIC biorepositories \nThe functioning of biorepositories requires stability and continuity of the LIMS. In addition to the buying of commercial licenses, some unforeseen expenses were encountered in the implementation of the commercial LIMS in H3Africa, including training costs for staff and more complex hardware requirements, unlike open-source LIMS. In H3Africa, it was possible to budget for these contingencies, but these costs may become prohibitive for smaller biorepositories outside of a funded mechanism. In such cases, potential creative solutions include forming consortia with similar facilities to buy a multi-user license. This approach has the added advantage of enabling a shared forum for dealing with other problems. However, since each biorepository has separate views, specific role-based security would be implemented for each user's collections. It may also only be necessary to acquire specific modules within a commercial package to keep the costs lower.\nRetaining and training staff to use LIMS have presented some challenges for H3Africa. Some strategies were used, including actively searching for personnel with experience, ensuring that the LIMS manager feels integrated into the biorepository management structure, and training junior staff to make sure that there is an adequate succession plan. In addition, including training clauses within purchasing agreements have mitigated some of the risk of purchasing complex licenses, but ongoing training represents an essential need and should be part of a biorepository's quality management system. Discussion of formulating in-house training material for LIMS support is ongoing across all three H3Africa Biorepositories for them to stay aligned with training and SOPs.\n\nIT infrastructure \nIn addition to high acquisition costs, the commercial LIMS requires an IT infrastructure to fully support their function. IT infrastructure is a combined set of the hardware (e.g., servers and computers), software (e.g., operating software), and network systems required to deploy and support the LIMS. As biospecimen numbers grow, there is a corresponding decrease in LIMS functionality such as very slow loading and processing speeds, which is linked to the supporting IT infrastructure.[8] Therefore, a LIMS infrastructure should have the ability to be scaled to meet the needs of the community it serves. Institutions that are unable to support a biorepository LIMS on their own should team with other institutions to develop and deploy a shared biorepository LIMS infrastructure. By using a shared infrastructure, each participating institution could support components of the infrastructure independently, while also collectively managing the entire biorepository architecture. This will not only give an economical benefit but also offer an environment for harmonizing complex, but still critical, components of a LIMS, such as structured data files and data models, as well as standards for data transmission.\n\nUser support services \nCommercial LIMS often needs user support services such as customization, implementation assistance, annual licenses, maintenance, and updates, although the need for these services may vary over time, especially as users become experienced with the system. User support can be provided in several forms, including telephonic support and on-site support. There is an additional cost to get access to such services, which may be minimized by conducting thorough initial training and license negotiation. Many commercial LIMS vendors have no support networks or offices in Africa. This increases the costs of user support services because of airfare, accommodation, and other attendant costs to access the services. In such circumstances, African biorepositories should try to use other remote-access technology to get access to support, such as public IP addresses that enable external access and manipulation of the LIMS. Unlike commercial LIMS, open-source LIMS do not have user support services and the user must troubleshoot locally, which can be challenging or impossible based on technical capacity.\n\nLIMS harmonization in the H3Africa biorepositories \nBefore the H3Africa program, each biorepository had its own LIMS that met their needs. Since the commencement of the program, the biorepositories have either acquired new LIMS or upgraded their existing systems following a thorough LIMS assessment program. The biorepositories conducted a harmonization exercise to ensure interoperability of the LIMS across the three sites. Pilot studies conducted between H3Africa biorepositories integrated data sharing and importation protocols through a pilot biospecimen and data exchange. Data exchange harmonization is essential if biorepositories are to work efficiently in networks such as H3Africa to support population genomics studies. Biorepositories need to define an agreed sharable set of data and data format for harmonization and interoperability to ease exchange. Pilot projects with virtual data transfer protocols were undertaken successfully, suggesting that this harmonization has been effective.\n\nLIMS sustainability \nThe biorepositories should develop strategies for long-term LIMS sustainability. Reliable and adequate sources of funding are keys to sustainability of LIMS. Commercial LIMS vendors require the payment of annual support fees unlike open-source LIMS. The risk of liquidation needs to be considered as it impacts LIMS support and maintenance and the long-term viability of the biorepository. Biorepositories should implement cost recovery measures for users to ensure sustainability.\n\nConclusion \nDeveloping a state-of-the-art biorepository requires much capacity and staff development, including acquisition of formal training, equipment, and software. Key among the biorepository infrastructure needs is a LIMS. Choosing a LIMS in low- and middle-income countries requires careful consideration of the various factors that could affect its successful and sustainable deployment and use. H3Africa biorepositories operating in a consortium have highlighted key factors that affect recommendations for successful LIMS choice and implementation.\n\nAcknowledgements \nThe H3Africa Consortium Biorepositories are funded by the U.S. National Institutes of Health grants UH3HG007008, UH3HG007051, UH3HG007438, and UH2HG007092 of the H3Africa Common Fund Initiative. The authors thank Autoscribe Informatics (Berkshire, UK), who provided the Laboratory Information Management System (LIMS) User Checklist used in the LIMS evaluation process. The following Investigators were members of the H3Africa Biorepository PI Working Group: Jennifer Troyer and Louise Wideroff, National Institutes of Health, Washington, District of Columbia. Sue Penno, Biorepository Consultant, National Institutes of Health, Washington, District of Columbia. Christine M. Beiswanger, Coriell Institute for Medical Research, Institute of Human Virology-Nigeria, Abuja, Nigeria.\n\nAuthor disclosure statement \nNo conflicting financial interests exist.\n\nReferences \n\n\n\u2191 H3Africa Consortium; Rotimi, C.; Abayomi, A. et al. (2014). \"Enabling the genomic revolution in Africa\". Science 344 (6190): 1346\u20138. doi:10.1126\/science.1251546. PMC PMC4138491. PMID 24948725. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4138491 .   \n\n\u2191 Human Heredity and Health in Africa (January 2011). \"Harnessing Genomic Technologies Toward Improving Health in Africa: Opportunities and Challenges\" (PDF). H3Africa Consortium. http:\/\/h3africa.org\/images\/PDF\/h3africa_whitepaper.pdf .   \n\n\u2191 3.0 3.1 3.2 Campbell, L.D.; Betsou, F.; Garcia, D.L. et al. (2012). \"Development of the ISBER Best Practices for Repositories: Collection, Storage, Retrieval and Distribution of Biological Materials for Research\". Biopreservation and Biobanking 10 (2): 232-233. doi:10.1089\/bio.2012.1025.   \n\n\u2191 4.0 4.1 Vaught, J.; Campbell, L.D.; Betsou, F. et al. (2012). \"The ISBER Best Practices: Insight from the editors of the third edition\". Biopreservation and Biobanking 10 (2): 76-78. doi:10.1089\/bio.2012.1024. PMID 24844903.   \n\n\u2191 Pitt, K.; Betsou, F. (2012). \"The ISBER Best Practices Self Assessment Tool (SAT): Lessons learned after three years of collecting responses\". Biopreservation and Biobanking 10 (6): 548-9. doi:10.1089\/bio.2012.1064. PMC PMC4046752. PMID 24845142. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4046752 .   \n\n\u2191 Betsou, F.; Luzergues, A.; Carter, A. et al. (2007). \"Towards norms for accreditation of biobanks for human health and medical research: Compilation of existing guidelines into an ISO certification\/accreditation norm-compatible format\". Quality Assurance Journal 11 (3\u20134): 221-294. doi:10.1002\/qaj.425.   \n\n\u2191 Landgraf, K.M.; Kakkar, R.; Meigs, M. et al. (2016). \"Open-source LIMS in Vietnam: The path toward sustainability and host country ownership\". International Journal of Medical Informatics 93: 92\u2013102. doi:10.1016\/j.ijmedinf.2016.06.010. PMID 27435952.   \n\n\u2191 Henricks, W.H. (2016). \"Laboratory information systems\". Clinics in laboratory medicine 36 (1): 1\u201311. doi:10.1016\/j.cll.2015.09.002. PMID 26851660.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\">https:\/\/www.limswiki.org\/index.php\/Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 15 May 2017, at 20:19.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 781 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","021382bc6a00c9f9a06aef062ae61eaa_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries_The_H3Africa_experience_and_lessons_learned skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Selecting a laboratory information management system for biorepositories in low- and middle-income countries: The H3Africa experience and lessons learned<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Biorepositories in Africa need significant infrastructural support to meet International Society for Biological and Environmental Repositories (ISBER) Best Practices to support population-based genomics research. ISBER recommends a biorepository information management system which can manage workflows from biospecimen receipt to distribution. The H3Africa Initiative set out to develop regional African biorepositories where Uganda, Nigeria, and South Africa were successfully awarded grants to develop the state-of-the-art biorepositories. The biorepositories carried out an elaborate process to evaluate and choose a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management system<\/a> (LIMS) with the aim of integrating the three geographically distinct sites. In this article, we review the processes, African experience, and lessons learned, and we make recommendations for choosing a biorepository LIMS in the African context.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>A biorepository requires the linkage of high-quality material to data housed in a laboratory information management system (LIMS), which tracks each sample. A founding principle of H3Africa is to make sure that DNA (and possibly other biological material) would be properly stored in biorepositories for future research purposes.<sup id=\"rdp-ebb-cite_ref-H3AfricaEnabling14_1-0\" class=\"reference\"><a href=\"#cite_note-H3AfricaEnabling14-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-H3AfricaHarnessing11_2-0\" class=\"reference\"><a href=\"#cite_note-H3AfricaHarnessing11-2\" rel=\"external_link\">[2]<\/a><\/sup> The value of this material is partly determined by the associated phenotypic data. Management of this data and sample tracking in compliance with national and international best practices and ethical guidelines requires a refined data management system.<sup id=\"rdp-ebb-cite_ref-CampbellDevelop12_3-0\" class=\"reference\"><a href=\"#cite_note-CampbellDevelop12-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VaughtTheISBER_12_4-0\" class=\"reference\"><a href=\"#cite_note-VaughtTheISBER_12-4\" rel=\"external_link\">[4]<\/a><\/sup> According to the International Society for Biological and Environmental Repositories (ISBER) Best Practices, a computer-based inventory system is necessary to track the location and annotation of every specimen in the biorepository.<sup id=\"rdp-ebb-cite_ref-CampbellDevelop12_3-1\" class=\"reference\"><a href=\"#cite_note-CampbellDevelop12-3\" rel=\"external_link\">[3]<\/a><\/sup> The system should also track significant events during a sample's existence from collection to destruction, including sample thaws, receipt and\/or processing delays, processing, transfer of the sample within the repository, specimen distribution and return, and destruction.<sup id=\"rdp-ebb-cite_ref-CampbellDevelop12_3-2\" class=\"reference\"><a href=\"#cite_note-CampbellDevelop12-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VaughtTheISBER_12_4-1\" class=\"reference\"><a href=\"#cite_note-VaughtTheISBER_12-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PittTheISBER12_5-0\" class=\"reference\"><a href=\"#cite_note-PittTheISBER12-5\" rel=\"external_link\">[5]<\/a><\/sup> These ISBER Best Practices, as well as others from around the world, are under consideration to codify International Organization for Standardization norms as the basis for a new international accreditation program for biorepositories.<sup id=\"rdp-ebb-cite_ref-BetsouTowards07_6-0\" class=\"reference\"><a href=\"#cite_note-BetsouTowards07-6\" rel=\"external_link\">[6]<\/a><\/sup> There are several factors that hinder successful implementation of a biorepository LIMS in Africa. This article discusses the H3Africa Biorepositories experiences and provides recommendations for the process of evaluating, choosing, and deployment of a sustainable biorepository LIMS in low- and middle-income countries (LMIC).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<p>In 2012 and 2013, the National Institutes of Health-funded H3Africa Program established three regional biorepositories in Africa: Uganda, South Africa, and Nigeria. The biorepositories were developed to archive and facilitate international scientific access to well-curated biospecimens collected by research projects in the H3Africa Consortium Program. To facilitate international access to biospecimens, it was desirable that the biorepositories set up an interoperable LIMS for data integration and exchange. At the time of funding, each of the three biorepositories had a LIMS for their current capacity. The biorepositories adopted a user requirements checklist provided by <a href=\"https:\/\/www.limswiki.org\/index.php\/Autoscribe_Informatics,_Inc.\" title=\"Autoscribe Informatics, Inc.\" class=\"mw-redirect wiki-link\" target=\"_blank\" data-key=\"a187eaeb93bae7889be44009f869fc9c\">Autoscribe Informatics<\/a> (Berkshire, UK) for the LIMS evaluation. The checklist consists of 15 key user requirements, including, but not limited to, general system information, system configuration and customization, user access and security, functionality, static data tables, reporting system, and system support. Six commercial LIMS companies were invited to respond to the questions and requirements in the evaluation tool. Two open-source LIMS were self-evaluated by the H3Africa biorepositories in Uganda and South Africa. The returned results from three commercial LIMS vendors were combined for analysis and discussed during conference calls and online correspondence. Two biorepository consultants provided advice during the evaluation process. The technical description of how the commercial and open-source LIMS compared with each other is beyond the scope of this article. However, we describe the experience and lessons drawn from the LIMS evaluation process from an African\/LIMC perspective.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"LIMS_selection_at_the_H3Africa_biorepositories\">LIMS selection at the H3Africa biorepositories<\/span><\/h3>\n<p>Following the evaluation process, the H3Africa biorepositories made decisions on their biorepository LIMS. The Institute of Human Virology Nigerian\u2013H3Africa Biorepository (I-HAB) in Nigeria decided to upgrade the current LIMS with the latest software updates from the vendor. The Integrated Biorepository of H3Africa Uganda (IBRH3AU) in Uganda decided to acquire a new LIMS and migrate the data from the old platform to the new LIMS. The Clinical Laboratory Services\u2013H3Africa Biorepository (CLS) in South Africa decided to acquire a new LIMS platform in addition to the existing LIMS. Both CLS and IBRH3AU chose the same new LIMS platforms. I-HAB was convinced that an upgrade was necessary and sufficient to achieve the interoperable requirements of the three program biorepositories.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Checklist_for_choosing_a_LIMS\">Checklist for choosing a LIMS<\/span><\/h3>\n<p>The H3Africa project specifications required that the three biorepositories' workflows would be harmonized through interoperable LIMS to enable data integration and exchange. Following the evaluation process, the following key elements were considered while choosing a LIMS for the project: (1) customizability and usability; (2) interoperability with other LIMS; (3) access to revisions, updates, patches, and maintenance releases; (4) cost and access to technical support services; (5) maintenance and associated costs; (6) multi-user\/site support; (7) robustness to handle large volumes of sample information; (8) security systems (audit trail, user roles, and privileges, etc.); and (9) type of (open-source or commercial) LIMS. Table 1 represents a summarized checklist of the elements we considered to be key while choosing a biorepository LIMS. A detailed checklist can be accessed via <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/biorepository.h3africa.org\/\" target=\"_blank\">biorepository.h3africa.org<\/a> website.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> Summarized checklist to consider while choosing a biorepository laboratory information management system. A detailed checklist can be accessed via <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/biorepository.h3africa.org\/\" target=\"_blank\">biorepository.h3africa.org<\/a>. CFR = Code of Federal Regulations; LIMS = laboratory information management system\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Category\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Considerations\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LIMS general information\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002What development tools have been used in the creation of the LIMS (e.g., Microsoft C#)?<br \/>\u2022\u2002Is the system scalable and therefore suitable for small organizations as well as global corporations?<br \/>\u2022\u2002What are the system requirements?<br \/>\u2022\u2002Does the system allow concurrent operation of three or more database groups in the standard system?\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System configuration\/customization\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002Is your system configurable so that it can exactly meet our requirements? If yes, what are the requirements? If no, what are the alternatives?<br \/>\u2022\u2002Is the system configurable by the nonprogrammer and without use of special languages?<br \/>\u2022\u2002Are user configuration changes supported by the vendor? Is there any extra cost?\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">User access\/security\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002What is the security access system in detail?<br \/>\u2022\u2002Does your system password requirements comply with 21CFR Part 11?\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Functionality\u2014sample\/work registration\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002Does your system support the following types of registration: Single sample; Single sample with copy feature; Batch registration with\/without copy feature; Bar-code support; Registration templates; Registration from external system and\/or scheduler; Spreadsheet style registration, including data capture from Excel; Can details of submitters and sample types be viewed from the registration screen; Can reports (e.g., worksheets and labels be automatically generated)?<br \/>\u2022\u2002Does system allow fields to be populated automatically by defining a default value within the system which can be overtyped by a user with suitable authority?\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Functionality\u2014sample receipt\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002Is a sample receipt function provided with the system? Useful when some work is preregistered before the availability of the samples. The receipt function is needed to track the arrival of the samples. Useful in checking whether all expected samples arrive in the laboratory and the time interval between registration and receipt for each sample can be measured. The sample turnaround time in the laboratory should be measured from the receipt date and time.<br \/>\u2022\u2002Does sample receipt allow for single sample, multiple sample, batch sample, and global sample receipt utilizing bar-codes where needed?\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Functionality\u2014sample preparation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002Is a sample preparation function provided with the system? This would be used to indicate that samples must complete the preparation stage before they are ready for testing.<br \/>\u2022\u2002Does sample preparation allow for single sample, multiple sample, batch sample, and global sample preparation?\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Functionality\u2014result entry\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002Does system include: Result entry by sample\u2014entry of any\/all test results for a single sample; Result entry by test\u2014entry of one test result for multiple samples; Result entry for multiple samples and multiple tests in spreadsheet style; Result import from a variety of sources including files and instruments; Viewing of results previously entered using same selection criteria as for selection of samples for result entry; Viewing of test status.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reporting\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002What reporting tools does the system support?<br \/>\u2022\u2002Is event triggered reporting\u2014reports generated by sample status change, for example\u2013included?\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System support\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002If I have a support question can I telephone a help-desk and immediately talk to a technical person familiar with my system? If yes, where would this person be located and what time can I call them? If no, what support scheme is in place?<br \/>\u2022\u2002Does the vendor have global coverage if relevant? If so what are the support center locations?<br \/>\u2022\u2002Are new version upgrades supplied to customers at no cost?<br \/>\u2022\u2002How much effort is typically needed for the implementation of an upgrade and what do I have to pay?<br \/>\u2022\u2002Will an upgrade preserve my configuration or custom code as well as my data?<br \/>\u2022\u2002Do you guarantee that we will always be able to upgrade to the next version?\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Miscellaneous\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022\u2002Does system allow storage of BLOB files, for example, pictures and documents (Consent forms) associated with a sample or a test?<br \/>\u2022\u2002Can a document management capability be fully integrated within the system?<br \/>\u2022\u2002Describe the sample tracking features of your LIMS. How is this used to monitor inventory for example?<br \/>\u2022\u2002Can I have separate databases for different departments within my organization (they may be in different locations)? Can these databases be configured differently?\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Commercial_versus_open-source_LIMS\">Commercial versus open-source LIMS<\/span><\/h3>\n<p>Commercial LIMS are systems whose source codes are developed for sale and need authorization from vendors for licensed use. Open-source LIMS are systems whose source codes are available for distribution at no cost.<sup id=\"rdp-ebb-cite_ref-LandgrafOpen16_7-0\" class=\"reference\"><a href=\"#cite_note-LandgrafOpen16-7\" rel=\"external_link\">[7]<\/a><\/sup> During the H3Africa biorepository LIMS evaluation phase, it was decided to implement a commercial LIMS in support of the H3Africa biospecimen collections. Commercial LIMS are much more expensive upfront than open-source LIMS, are less flexible for end-user adaptations, but do not need local expertise for support. Despite some features of modified open-source LIMS that might seem more applicable to the African setting, there were significant concerns about the stability of such systems and the lack of standardization. In addition, it was clear that the adaptation and maintenance of such a system would need highly specialized staff at three biorepositories and that this approach could create differences among the biorepositories, which could cause potential delays in interbiorepository transfer of data and material.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Financing_of_LIMS_in_the_LMIC_biorepositories\">Financing of LIMS in the LMIC biorepositories<\/span><\/h3>\n<p>The functioning of biorepositories requires stability and continuity of the LIMS. In addition to the buying of commercial licenses, some unforeseen expenses were encountered in the implementation of the commercial LIMS in H3Africa, including training costs for staff and more complex hardware requirements, unlike open-source LIMS. In H3Africa, it was possible to budget for these contingencies, but these costs may become prohibitive for smaller biorepositories outside of a funded mechanism. In such cases, potential creative solutions include forming consortia with similar facilities to buy a multi-user license. This approach has the added advantage of enabling a shared forum for dealing with other problems. However, since each biorepository has separate views, specific role-based security would be implemented for each user's collections. It may also only be necessary to acquire specific modules within a commercial package to keep the costs lower.\n<\/p><p>Retaining and training staff to use LIMS have presented some challenges for H3Africa. Some strategies were used, including actively searching for personnel with experience, ensuring that the LIMS manager feels integrated into the biorepository management structure, and training junior staff to make sure that there is an adequate succession plan. In addition, including training clauses within purchasing agreements have mitigated some of the risk of purchasing complex licenses, but ongoing training represents an essential need and should be part of a biorepository's quality management system. Discussion of formulating in-house training material for LIMS support is ongoing across all three H3Africa Biorepositories for them to stay aligned with training and SOPs.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"IT_infrastructure\">IT infrastructure<\/span><\/h3>\n<p>In addition to high acquisition costs, the commercial LIMS requires an IT infrastructure to fully support their function. IT infrastructure is a combined set of the hardware (e.g., servers and computers), software (e.g., operating software), and network systems required to deploy and support the LIMS. As biospecimen numbers grow, there is a corresponding decrease in LIMS functionality such as very slow loading and processing speeds, which is linked to the supporting IT infrastructure.<sup id=\"rdp-ebb-cite_ref-HenricksLab16_8-0\" class=\"reference\"><a href=\"#cite_note-HenricksLab16-8\" rel=\"external_link\">[8]<\/a><\/sup> Therefore, a LIMS infrastructure should have the ability to be scaled to meet the needs of the community it serves. Institutions that are unable to support a biorepository LIMS on their own should team with other institutions to develop and deploy a shared biorepository LIMS infrastructure. By using a shared infrastructure, each participating institution could support components of the infrastructure independently, while also collectively managing the entire biorepository architecture. This will not only give an economical benefit but also offer an environment for harmonizing complex, but still critical, components of a LIMS, such as structured data files and data models, as well as standards for data transmission.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"User_support_services\">User support services<\/span><\/h3>\n<p>Commercial LIMS often needs user support services such as customization, implementation assistance, annual licenses, maintenance, and updates, although the need for these services may vary over time, especially as users become experienced with the system. User support can be provided in several forms, including telephonic support and on-site support. There is an additional cost to get access to such services, which may be minimized by conducting thorough initial training and license negotiation. Many commercial LIMS vendors have no support networks or offices in Africa. This increases the costs of user support services because of airfare, accommodation, and other attendant costs to access the services. In such circumstances, African biorepositories should try to use other remote-access technology to get access to support, such as public IP addresses that enable external access and manipulation of the LIMS. Unlike commercial LIMS, open-source LIMS do not have user support services and the user must troubleshoot locally, which can be challenging or impossible based on technical capacity.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"LIMS_harmonization_in_the_H3Africa_biorepositories\">LIMS harmonization in the H3Africa biorepositories<\/span><\/h3>\n<p>Before the H3Africa program, each biorepository had its own LIMS that met their needs. Since the commencement of the program, the biorepositories have either acquired new LIMS or upgraded their existing systems following a thorough LIMS assessment program. The biorepositories conducted a harmonization exercise to ensure interoperability of the LIMS across the three sites. Pilot studies conducted between H3Africa biorepositories integrated data sharing and importation protocols through a pilot biospecimen and data exchange. Data exchange harmonization is essential if biorepositories are to work efficiently in networks such as H3Africa to support population genomics studies. Biorepositories need to define an agreed sharable set of data and data format for harmonization and interoperability to ease exchange. Pilot projects with virtual data transfer protocols were undertaken successfully, suggesting that this harmonization has been effective.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"LIMS_sustainability\">LIMS sustainability<\/span><\/h3>\n<p>The biorepositories should develop strategies for long-term LIMS sustainability. Reliable and adequate sources of funding are keys to sustainability of LIMS. Commercial LIMS vendors require the payment of annual support fees unlike open-source LIMS. The risk of liquidation needs to be considered as it impacts LIMS support and maintenance and the long-term viability of the biorepository. Biorepositories should implement cost recovery measures for users to ensure sustainability.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>Developing a state-of-the-art biorepository requires much capacity and staff development, including acquisition of formal training, equipment, and software. Key among the biorepository infrastructure needs is a LIMS. Choosing a LIMS in low- and middle-income countries requires careful consideration of the various factors that could affect its successful and sustainable deployment and use. H3Africa biorepositories operating in a consortium have highlighted key factors that affect recommendations for successful LIMS choice and implementation.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The H3Africa Consortium Biorepositories are funded by the U.S. National Institutes of Health grants UH3HG007008, UH3HG007051, UH3HG007438, and UH2HG007092 of the H3Africa Common Fund Initiative. The authors thank Autoscribe Informatics (Berkshire, UK), who provided the Laboratory Information Management System (LIMS) User Checklist used in the LIMS evaluation process. The following Investigators were members of the H3Africa Biorepository PI Working Group: Jennifer Troyer and Louise Wideroff, National Institutes of Health, Washington, District of Columbia. Sue Penno, Biorepository Consultant, National Institutes of Health, Washington, District of Columbia. Christine M. Beiswanger, Coriell Institute for Medical Research, Institute of Human Virology-Nigeria, Abuja, Nigeria.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Author_disclosure_statement\">Author disclosure statement<\/span><\/h2>\n<p>No conflicting financial interests exist.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-H3AfricaEnabling14-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-H3AfricaEnabling14_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">H3Africa Consortium; Rotimi, C.; Abayomi, A. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4138491\" target=\"_blank\">\"Enabling the genomic revolution in Africa\"<\/a>. <i>Science<\/i> <b>344<\/b> (6190): 1346\u20138. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.1251546\" target=\"_blank\">10.1126\/science.1251546<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4138491\/\" target=\"_blank\">PMC4138491<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24948725\" target=\"_blank\">24948725<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4138491\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4138491<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enabling+the+genomic+revolution+in+Africa&rft.jtitle=Science&rft.aulast=H3Africa+Consortium%3B+Rotimi%2C+C.%3B+Abayomi%2C+A.+et+al.&rft.au=H3Africa+Consortium%3B+Rotimi%2C+C.%3B+Abayomi%2C+A.+et+al.&rft.date=2014&rft.volume=344&rft.issue=6190&rft.pages=1346%E2%80%938&rft_id=info:doi\/10.1126%2Fscience.1251546&rft_id=info:pmc\/PMC4138491&rft_id=info:pmid\/24948725&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4138491&rfr_id=info:sid\/en.wikipedia.org:Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-H3AfricaHarnessing11-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-H3AfricaHarnessing11_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Human Heredity and Health in Africa (January 2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/h3africa.org\/images\/PDF\/h3africa_whitepaper.pdf\" target=\"_blank\">\"Harnessing Genomic Technologies Toward Improving Health in Africa: Opportunities and Challenges\"<\/a> (PDF). H3Africa Consortium<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/h3africa.org\/images\/PDF\/h3africa_whitepaper.pdf\" target=\"_blank\">http:\/\/h3africa.org\/images\/PDF\/h3africa_whitepaper.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Harnessing+Genomic+Technologies+Toward+Improving+Health+in+Africa%3A+Opportunities+and+Challenges&rft.atitle=&rft.aulast=Human+Heredity+and+Health+in+Africa&rft.au=Human+Heredity+and+Health+in+Africa&rft.date=January+2011&rft.pub=H3Africa+Consortium&rft_id=http%3A%2F%2Fh3africa.org%2Fimages%2FPDF%2Fh3africa_whitepaper.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CampbellDevelop12-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CampbellDevelop12_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-CampbellDevelop12_3-1\" rel=\"external_link\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-CampbellDevelop12_3-2\" rel=\"external_link\">3.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Campbell, L.D.; Betsou, F.; Garcia, D.L. et al. (2012). \"Development of the <i>ISBER Best Practices for Repositories: Collection, Storage, Retrieval and Distribution of Biological Materials for Research<\/i>\". <i>Biopreservation and Biobanking<\/i> <b>10<\/b> (2): 232-233. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fbio.2012.1025\" target=\"_blank\">10.1089\/bio.2012.1025<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+of+the+%27%27ISBER+Best+Practices+for+Repositories%3A+Collection%2C+Storage%2C+Retrieval+and+Distribution+of+Biological+Materials+for+Research%27%27&rft.jtitle=Biopreservation+and+Biobanking&rft.aulast=Campbell%2C+L.D.%3B+Betsou%2C+F.%3B+Garcia%2C+D.L.+et+al.&rft.au=Campbell%2C+L.D.%3B+Betsou%2C+F.%3B+Garcia%2C+D.L.+et+al.&rft.date=2012&rft.volume=10&rft.issue=2&rft.pages=232-233&rft_id=info:doi\/10.1089%2Fbio.2012.1025&rfr_id=info:sid\/en.wikipedia.org:Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VaughtTheISBER_12-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VaughtTheISBER_12_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-VaughtTheISBER_12_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vaught, J.; Campbell, L.D.; Betsou, F. et al. (2012). \"The ISBER Best Practices: Insight from the editors of the third edition\". <i>Biopreservation and Biobanking<\/i> <b>10<\/b> (2): 76-78. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fbio.2012.1024\" target=\"_blank\">10.1089\/bio.2012.1024<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24844903\" target=\"_blank\">24844903<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+ISBER+Best+Practices%3A+Insight+from+the+editors+of+the+third+edition&rft.jtitle=Biopreservation+and+Biobanking&rft.aulast=Vaught%2C+J.%3B+Campbell%2C+L.D.%3B+Betsou%2C+F.+et+al.&rft.au=Vaught%2C+J.%3B+Campbell%2C+L.D.%3B+Betsou%2C+F.+et+al.&rft.date=2012&rft.volume=10&rft.issue=2&rft.pages=76-78&rft_id=info:doi\/10.1089%2Fbio.2012.1024&rft_id=info:pmid\/24844903&rfr_id=info:sid\/en.wikipedia.org:Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PittTheISBER12-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PittTheISBER12_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pitt, K.; Betsou, F. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4046752\" target=\"_blank\">\"The ISBER Best Practices Self Assessment Tool (SAT): Lessons learned after three years of collecting responses\"<\/a>. <i>Biopreservation and Biobanking<\/i> <b>10<\/b> (6): 548-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fbio.2012.1064\" target=\"_blank\">10.1089\/bio.2012.1064<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4046752\/\" target=\"_blank\">PMC4046752<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24845142\" target=\"_blank\">24845142<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4046752\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4046752<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+ISBER+Best+Practices+Self+Assessment+Tool+%28SAT%29%3A+Lessons+learned+after+three+years+of+collecting+responses&rft.jtitle=Biopreservation+and+Biobanking&rft.aulast=Pitt%2C+K.%3B+Betsou%2C+F.&rft.au=Pitt%2C+K.%3B+Betsou%2C+F.&rft.date=2012&rft.volume=10&rft.issue=6&rft.pages=548-9&rft_id=info:doi\/10.1089%2Fbio.2012.1064&rft_id=info:pmc\/PMC4046752&rft_id=info:pmid\/24845142&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4046752&rfr_id=info:sid\/en.wikipedia.org:Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BetsouTowards07-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BetsouTowards07_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Betsou, F.; Luzergues, A.; Carter, A. et al. (2007). \"Towards norms for accreditation of biobanks for human health and medical research: Compilation of existing guidelines into an ISO certification\/accreditation norm-compatible format\". <i>Quality Assurance Journal<\/i> <b>11<\/b> (3\u20134): 221-294. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fqaj.425\" target=\"_blank\">10.1002\/qaj.425<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+norms+for+accreditation+of+biobanks+for+human+health+and+medical+research%3A+Compilation+of+existing+guidelines+into+an+ISO+certification%2Faccreditation+norm-compatible+format&rft.jtitle=Quality+Assurance+Journal&rft.aulast=Betsou%2C+F.%3B+Luzergues%2C+A.%3B+Carter%2C+A.+et+al.&rft.au=Betsou%2C+F.%3B+Luzergues%2C+A.%3B+Carter%2C+A.+et+al.&rft.date=2007&rft.volume=11&rft.issue=3%E2%80%934&rft.pages=221-294&rft_id=info:doi\/10.1002%2Fqaj.425&rfr_id=info:sid\/en.wikipedia.org:Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LandgrafOpen16-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LandgrafOpen16_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Landgraf, K.M.; Kakkar, R.; Meigs, M. et al. (2016). \"Open-source LIMS in Vietnam: The path toward sustainability and host country ownership\". <i>International Journal of Medical Informatics<\/i> <b>93<\/b>: 92\u2013102. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2016.06.010\" target=\"_blank\">10.1016\/j.ijmedinf.2016.06.010<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27435952\" target=\"_blank\">27435952<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open-source+LIMS+in+Vietnam%3A+The+path+toward+sustainability+and+host+country+ownership&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Landgraf%2C+K.M.%3B+Kakkar%2C+R.%3B+Meigs%2C+M.+et+al.&rft.au=Landgraf%2C+K.M.%3B+Kakkar%2C+R.%3B+Meigs%2C+M.+et+al.&rft.date=2016&rft.volume=93&rft.pages=92%E2%80%93102&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2016.06.010&rft_id=info:pmid\/27435952&rfr_id=info:sid\/en.wikipedia.org:Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HenricksLab16-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HenricksLab16_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Henricks, W.H. (2016). \"Laboratory information systems\". <i>Clinics in laboratory medicine<\/i> <b>36<\/b> (1): 1\u201311. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cll.2015.09.002\" target=\"_blank\">10.1016\/j.cll.2015.09.002<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26851660\" target=\"_blank\">26851660<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+information+systems&rft.jtitle=Clinics+in+laboratory+medicine&rft.aulast=Henricks%2C+W.H.&rft.au=Henricks%2C+W.H.&rft.date=2016&rft.volume=36&rft.issue=1&rft.pages=1%E2%80%9311&rft_id=info:doi\/10.1016%2Fj.cll.2015.09.002&rft_id=info:pmid\/26851660&rfr_id=info:sid\/en.wikipedia.org:Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181458\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.288 seconds\nReal time usage: 0.313 seconds\nPreprocessor visited node count: 8352\/1000000\nPreprocessor generated node count: 31731\/1000000\nPost\u2010expand include size: 70724\/2097152 bytes\nTemplate argument size: 27021\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 296.073 1 - -total\n 69.76% 206.544 1 - Template:Reflist\n 58.84% 174.199 8 - Template:Citation\/core\n 56.67% 167.795 7 - Template:Cite_journal\n 25.52% 75.566 1 - Template:Infobox_journal_article\n 24.75% 73.281 1 - Template:Infobox\n 15.80% 46.766 80 - Template:Infobox\/row\n 8.70% 25.749 14 - Template:Citation\/identifier\n 7.17% 21.240 1 - Template:Cite_web\n 4.65% 13.754 8 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10122-0!*!0!!en!*!* and timestamp 20181214181458 and revision id 30574\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned\">https:\/\/www.limswiki.org\/index.php\/Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","021382bc6a00c9f9a06aef062ae61eaa_images":[],"021382bc6a00c9f9a06aef062ae61eaa_timestamp":1544811297,"c1c21b43f0762acc3ad8147824a0e651_type":"article","c1c21b43f0762acc3ad8147824a0e651_title":"MASTR-MS: A web-based collaborative laboratory information management system (LIMS) for metabolomics (Hunter et al. 2017)","c1c21b43f0762acc3ad8147824a0e651_url":"https:\/\/www.limswiki.org\/index.php\/Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics","c1c21b43f0762acc3ad8147824a0e651_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:MASTR-MS: A web-based collaborative laboratory information management system (LIMS) for metabolomics\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nMASTR-MS: A web-based collaborative laboratory information management system (LIMS) for metabolomicsJournal\n \nMetabolomicsAuthor(s)\n \nHunter, A.; Dayalan, S.; De Souza, D.; Power, B.; Lorrimar, R.; Szabo, T.; Nguyen, T.; O'Callaghan, S.; Hack, J.;\r\nPyke, J.; Nahid, A.; Barrero, R.; Roessner, U.; Likic, V.; Tull, D.; Bacic, A.; McConville, M.; Bellgard, M.Author affiliation(s)\n \nMurdoch University, The University of Melbourne, The Australian Wine Research InstitutePrimary contact\n \nEmail: malcolmm at unimelb dot edu dot au -or- mbellgard at ccg dot murdoch dot edu dot auYear published\n \n2017Volume and issue\n \n13 (2)Page(s)\n \n14DOI\n \n10.1007\/s11306-016-1142-2ISSN\n \n1573-3890Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/link.springer.com\/article\/10.1007\/s11306-016-1142-2Download\n \nhttps:\/\/link.springer.com\/content\/pdf\/10.1007%2Fs11306-016-1142-2.pdf (PDF)\n\nContents\n\n1 Abstract \n\n1.1 Background \n1.2 Results \n1.3 Conclusion \n1.4 Keywords \n\n\n2 Introduction \n3 Materials and methods \n4 Results \n\n4.1 Node management system \n4.2 User management system \n\n4.2.1 Systems administrator \n4.2.2 Administrator \n4.2.3 Node representative \n4.2.4 Project leader \n4.2.5 Staff \n4.2.6 Client \n\n\n4.3 Quote management system \n4.4 Project management system \n\n4.4.1 Experiment details \n4.4.2 Access control\/roles \n4.4.3 Sample metadata \n4.4.4 Origin\/organs\/parts metadata \n4.4.5 Timeline\/treatment metadata \n4.4.6 Sample preparation \n4.4.7 Automatic sample class generation \n4.4.8 Runs \n4.4.9 Worklist generation \n\n\n4.5 Data management system \n\n4.5.1 Data acquisition and the DataSync Client \n4.5.2 Run progress \n\n\n\n\n5 Discussion \n\n5.1 Scope of MASTR-MS \n5.2 Comparison to similar software \n\n\n6 Conclusion \n7 Availability and requirements \n8 Acknowledgements \n9 Compliance with ethical standards \n\n9.1 Conflict of interest \n9.2 Ethical approval \n\n\n10 Supplementary material \n11 References \n12 Notes \n\n\n\nAbstract \nBackground \nAn increasing number of research laboratories and core analytical facilities around the world are developing high throughput metabolomic analytical and data processing pipelines that are capable of handling hundreds to thousands of individual samples per year, often over multiple projects, collaborations and sample types. At present, there are no laboratory information management systems (LIMS) that are specifically tailored for metabolomics laboratories that are capable of tracking samples and associated metadata from the beginning to the end of an experiment, including data processing and archiving, and which are also suitable for use in large institutional core facilities or multi-laboratory consortia as well as single laboratory environments.\n\nResults \nHere we present MASTR-MS, a downloadable and installable LIMS solution that can be deployed either within a single laboratory or used to link workflows across a multisite network. It comprises the Node Management System that can be used to link and manage projects across one or multiple collaborating laboratories; the User Management System which defines different user groups and privileges of users; the Quote Management System where client quotes are managed; the Project Management System in which metadata is stored and all aspects of project management, including experimental setup, sample tracking and instrument analysis, are defined; and the Data Management System that allows the automatic capture and storage of raw and processed data from the analytical instruments to the LIMS.\n\nConclusion \nMASTR-MS is a comprehensive LIMS solution specifically designed for metabolomics. It captures the entire lifecycle of a sample, starting from project and experiment design to sample analysis, data capture and storage. It acts as an electronic notebook, facilitating project management within a single laboratory or a multi-node collaborative environment. This software is being developed in close consultation with members of the metabolomics research community. It is freely available under the GNU GPL v3 license and can be accessed from https:\/\/muccg.github.io\/mastr-ms\/.\n\nKeywords \nMASTR-MS, metabolomics, LIMS, omics \n\nIntroduction \nMetabolomic approaches aim to detect and quantitate levels of all small molecules in a biological system and, together with other \"omic\" approaches, can be used to generate a systems-wide understanding of biological processes. Metabolomic approaches typically involve the use of advanced mass spectrometry and nuclear magnetic resonance (NMR) spectrometry platforms to maximize coverage of the chemically diverse metabolites that make up biological systems. In many cases, these analytical platforms are located in institutional and\/or national core facilities that offer a range of metabolomics capabilities to researchers.[1][2][3][4][5] These core facilities, as well as individual research groups with sophisticated metabolomics infrastructure and capability are faced with the challenge of tracking large numbers of samples and the associated metadata, and linking this information with the raw datasets generated by multiple analytical platforms, as well as processed down-stream data sets. Data handling extends beyond collection and curation of raw data, to the management of metadata that defines how the raw data is generated. Major funding agencies, such as Europe\u2019s Horizon 2020[5], the NIH[6], The Wellcome Trust[7] and Australia\u2019s NHMRC[8] have established data management plans that researchers are expected to follow in order to capture, store and share data generated by their grants. Scientific journals are also increasingly requesting that experimental data and metadata associated with metabolomics experiments are made available to the scientific community[9][10], leading to the establishment of data repositories, such as MetaboLights[11] and Metabolomics Workbench.[12]\nLIMS are software solutions that aim to manage the entire workflow of a laboratory. A number of LIMS have been developed or adapted from other applications for curating metabolomics experiments and data management (e.g., SetupX, Sesame). While these LIMS have features that allow capture of project metadata, experiments and samples, data storage, and data sharing, they exhibit a number of limitations around their capacity to accommodate different vendor instruments and have restricted functionalities to facilitate a collaborative configuration between geographically distributed laboratories. In this paper we present MASTR-MS, the first wholly functional, open-source LIMS solution specifically designed for metabolomics laboratories.\n\nMaterials and methods \nMASTR-MS runs as a Python[13] web application built on the Django[14] framework, utilising a PostgreSQL[15] or MySQL[16] relational database. MASTR-MS leverages the functionality of the Django framework for user management, users permissions and security. Django is a mature web framework and provides multiple security tools and mechanisms. For example, specific protection is provided against cross-site scripting (XSS), cross-site request forgery (CSRF), SQL injection and clickjacking. A security middleware is also used to enforce SSL\/HTTPS for all traffic. MASTR-MS is built using open-source components and communicates using open standards. The client side browser interface leverages Javascript and AJAX for fluid data display and submission, giving a user experience much like a desktop application, but with the flexibility of being available from any internet-connected location on any operating system, with no client-side download or installation.\nThe DataSync Client is a small desktop application that runs on an instrument\u2019s acquisition computer. This software constantly communicates with the MASTR-MS server and is responsible for transferring raw data from the acquisition computer to the MASTR-MS repository (Supplemental Fig. S9A). The DataSync Client is written in the Python programming language using the wxWidgets[17] GUI library and runs on Windows and Linux systems. Data is uploaded using the rsync protocol[18] and the libraries and plugins required for this are included in the installation package.\nAs the MASTR-MS server-side component is written in the Python 2.7 programming language, any operating system that has Python 2.7 available for running web applications with a web server can run the application. In practice, the application has only been tested on the Linux operating system and the Apache web server. For installation, operating system packages are available in RPM format for CentOS 6.5. Similarly, as the DataSync Client is also written in Python 2.7 it can run on any operating system that has Python 2.7 available. However it is typically installed on a Windows platform with a connected analytical instrument. For this reason, the DataSync Client is distributed as a Windows executable (.exe) installer. The DataSync Client application is also self-updating by means of a user option to upgrade to a newer version if available.\n\nResults \nMASTR-MS is a web-based LIMS solution for metabolomics laboratories. The different modules of MASTR-MS allow users to: \n\n Track all metabolomics samples and associated meta-, analytical- and processed data sets. This starts from the capture of client\/collaborator communication; the establishment of new projects, experimental design and sample definitions; and the automatic capture of raw data generated by the instruments.\n Develop an electronic notebook, where users record all relevant information about projects and experiments in MASTR-MS, thus allowing multiple users to work on the same project.\n Methodically manage the vast amount of data generated by the analytical instruments, by associating it with the project, experiment and sample details.\n Facilitate collaboration between geographically distributed laboratories through the sharing of projects and experiment data.\nMASTR-MS is equally suited for use in either a large core facility or single-\/multi-laboratory environment. Thus, both large national facilities and small individual laboratories would equally benefit from using MASTR-MS.\nMASTR-MS comprises five major modules: (1) the Node Management System, (2) the User Management System, (3) the Quote Management System, (4) the Project Management System and (5) the Data Management System. Figure 1 shows the workflow of MASTR-MS using the different functionalities and features. These functions are described in detail below. The user is initially connected to the dashboard when they first log into MASTR-MS, and the available functions are tailored to the level of access of the user. The dashboard gives an \u201cat-a-glance\u201d summary of recent activity on the site and items requiring attention. Depending on the user\u2019s status\/level of access, the dashboard shows pending user requests, quotes requiring attention, recently created \/ modified projects, and recently created \/ modified experiments.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Overview of MASTR-MS system workflow\n\n\n\nNode management system \nThis module allows the addition of multiple laboratories to be part of a single MASTR-MS network. For example, a group of geographically dispersed laboratories can have a single deployment of MASTR-MS and share projects and experiments. Such a setup would be established by the module through the generation of different nodes. On the other hand, MASTR-MS can be used within a single laboratory environment in which this module would comprise a single node.\n\nUser management system \nThis module defines the different user groups used in MASTR-MS. Each user group has different privileges and permissions to access the different functionalities of MASTR-MS. In addition, this module allows the generation and management of users of the system. MASTR-MS has several user groups.\n\nSystems administrator \nThis user group has access to all functionalities of MASTR-MS. There would normally be one assigned Systems Administrator who would act as the query point for all other users accessing the system, although it is possible to have more than one Systems Administrator. The Systems Administrator has a Laboratory Name assigned to their account (like all other users), allowing a nominated user, usually a member of the organization\/laboratory that is hosting the project to act as the Systems Administrator. The Systems Administrator can add new users to the system, assign user groups to any users in any laboratory, edit details of users and delete users of any laboratory.\n\nAdministrator \nThis user group has full access to all projects, experiments and experimental data, user accounts and quotes within MASTR-MS, regardless of node. This user group allows selected users to view all projects and experiments across different nodes, allowing seamless sharing and collaboration of data across nodes. Where multiple laboratories have a single MASTR-MS deployment, but prefer not to share projects and experiments, no users would be assigned the Administrator role.\n\nNode representative \nThis user group has full access to quotes for their node and are the preferred contact for quotes and projects run by this node (detailed more in the \"Quote management system\" section). In a multi-node setup there would typically be at least one user assigned to this group per node.\n\nProject leader \nThis user group is able to create new projects and experiments for their node. Additionally, this group is able to assign staff to specific projects and experiments.\n\nStaff \nUsers of this group are able to participate in the projects and experiments for their node.\n\nClient \nAll other users of the system are clients. This group has no privileges other than viewing the progress of projects to which they have been assigned.\nAny user of the system can update their own user record and change their password at any time.\n\nQuote management system \nThis module was designed specifically for core facilities that provide metabolomic services to client researchers. Potential clients can request a pricing quote for running samples of an experiment through the quote request system without having to sign up for an account. At a nominated stage, clients are required to register in MASTR-MS by completing a short information dialog box. This module allows collection of contact details and information about the nature of the request. Files in various formats can be attached to this module. In a multi-node facility, the user can either direct their quote to a specific node with relevant expertise or they can select \"Don\u2019t Know\" to have all the Node Representatives alerted.\nQuote requests made by clients and collaborators that are made through the system are tracked and marked if they have not been attended to yet, so that Node Representatives can quickly see new quotes which require attention. Quotes can only be seen by members of the node to which they were sent, unless the \"Don\u2019t Know\" option was selected. Node Representatives are able to forward quotes to other nodes if required. The Node Representatives can then begin a dialogue with the potential client and with their team, clarifying the task, and providing formal quotes, attached as PDFs if necessary. Each step of the communications process is time-stamped and tracked within this module. The quote requests and any resulting quotes would eventually be associated with a project and experiment through a selection option in the Experimental Design stage. All documentation relating to the project, including the client and quote issued for the project, along with the project and experimental setup, is thus kept together.\n\nProject management system \nThis module allows the management of projects, experiments, and samples as well as the creation of analytical sample runs. As detailed above, users of different user groups are able to create projects and experiments. When a project is created by either a MASTR-MS Administrator or Project Leader, it can be linked to a specific client from the user list. This allows the client to monitor how the project is progressing. Assigning a Project Manager to the project allows those users to manage all aspects of a project, experiment creation and further access control on an experiment-by-experiment basis (Supplemental Fig. S4). As sample metadata is linked to all experiments within MASTR-MS, sample classes and\/or individual samples can be organized into groups and subsequently analyzed on an instrument.\n\nExperiment details \nThe Experiment Status defaults to \"New\" when first opened, and all experiment metadata is captured in this field (Supplemental Fig. S5A). Once the experiment design has been completed, the Project Manager can change the setting to \"Designed\" to prevent further changes. The experiment can also be linked to a formal quote that has been previously entered in the quotes system, and if needed, can be assigned an internal job number.\n\nAccess control\/roles \nUsers can be assigned to an experiment, giving them access to edit the experimental workflow and create samples and runs. Client users can also be added here, giving them access to project progress information (Supplemental Fig. S5B).\n\nSample metadata \nMASTR-MS uses sample metadata in order to generate sample classes, which can then be populated with individual samples (Supplemental Fig. S5C).\n\nOrigin\/organs\/parts metadata \nThe first metadata category is the Origin field, which contains information on sample origin and preparation (Supplemental Fig. S5D). Different metadata fields are available depending on whether the source is Microbial, Plant, Animal, Human, Synthetic, or Other.\n\nTimeline\/treatment metadata \nMASTR-MS also accepts time course and treatment metadata, where samples have been collected over multiple time points, or after different experimental treatments. The Origin, Timeline, and Treatment fields are then used to automatically generate sample classes.\n\nSample preparation \nMASTR-MS allows an upload of a standard operating procedure (SOP) document to be associated with an experiment. Multiple SOPs can be uploaded and additional notes recorded for each. A SOP is linked with methods used during runs at the time of setting up a run. The SOP is linked at the experiment level, and the option of choosing methods is provided under the runs level. This is to incorporate the option where a user would like to run multiple methods during a run (either by resampling the same vial or from a different vial).\n\nAutomatic sample class generation \nBased on the metadata entered in the Origin, Timeline, and Treatment steps, sample classes are automatically generated based on permutations of the available metadata (Supplemental Fig. S7A). If abbreviations have been provided for a particular metadata category, these will be used during sample class generation. Samples can then be created in each sample class.\nSamples can then be viewed and collected together to form a run on a designated analytical instrument platform (Supplemental Fig. S7B). Additional sample information can be imported via CSV and exported from MASTR-MS in the same way. Samples can be randomized before putting them into a run if desired.\n\nRuns \nSelected samples are added to a new or existing run by clicking the \"Add Selected Samples to Run\" button. This will display a dialog allowing the user to add either the samples to a new run or to any previous run which is still unlocked for editing (Supplemental Fig. S8A). Runs continue to be unlocked as long as a worklist has not yet been generated for them. Locked runs can be edited and reused if needed using the \u201cRun Cloning\u201d feature, which will duplicate the run data into a new unlocked run.\n\nWorklist generation \nThe goal of run configuration is to streamline sample analysis and generate instrument worklists in a convenient and flexible manner. After sample data has been added to a run, the order and sequencing of additional run elements (Sweeps, Solvents, etc.) can be added via the Rules Generator.\nThe Rules Generator provides a customizable set of steps (rules) which dictate how worklists are built. It consists of a Start Block, Sample Block, and End Block, each of which allows the insertion of non-sample components into the worklist. These include Pooled Biological QC, Sweep, Reagent Blank, Solvent Blank and Pure Standard.\nThe sample block, containing the experiment samples, allows n components to be inserted every m samples, in random or position order (Supplemental Fig. S8B). Once all three blocks have been designed, the Rule Generator can be enabled, disabling further editing and making the rule available for inclusion in run worklist generation. Rule Generators can be restricted to use by a single user, an entire node, or everybody on the system. Enabled Rule Generators can be cloned in order to generate a new version, which can then be extended and modified.\nTo generate a worklist within a run, the user selects an instrument (configured and made available by Administrators) and a Rule Generator, if needed, and clicks the \"Generate Worklist\" button. Once the worklist is generated, further modification of the run is not possible. The specific worklist format is customizable by site administrators to provide flexibility among various instrument models. Once the worklist is generated, it can be used with the instrument to automate the raw data collection process.\n\nData management system \nThis module facilitates the capture and storage of raw data produced by the instruments. The raw data is captured by the DataSync Client as detailed below and is linked to associated project and experiment details. In addition, post-processed data and any other related files such as presentations, reports and papers can be linked to the data.\n\nData acquisition and the DataSync Client \nThe DataSync Client allows data to be transferred from connected instruments at nominated frequencies and will run in the background of the acquisition computers as an icon in the System Tray. The software is fully integrated with the MASTR-MS web application. When data synchronization is requested, either scheduled or manually, the DataSync Client communicates with the MASTR-MS system to query all incomplete experiment runs which have been configured for the connected instrument. It then searches the acquired data for required files and transmits them to the MASTR-MS repository via a configurable rsync transport, allowing compression and check-summing for efficient data transfer. The configuration options for individual DataSync Nodes are fully configurable via the MASTR-MS administration interface.\nTo enable DataSync Client uploads on the instrument, the user simply selects the connected instrument from the list which has been configured on the MASTR-MS system and enters the Rsync username which they have been assigned (Supplemental Fig. S9B). OpenSSH Public Keys can be uploaded to the MASTR-MS system for secure password-less usage, which allows the client to run seamless automated data uploads without need for operator intervention.\nThe DataSync Client can also be configured with some advanced options. Data archival allows the raw sample data to be automatically replicated in a specified location (e.g., on another hard disk) once confirmation of upload has been achieved, allowing the original data to be deleted if desired.\nThe software can also be forced to re-synchronize experiment data that has been marked as complete in case the need arises (Supplemental Fig. S9C).\n\nRun progress \nAs data is synced with the MASTR-MS system, run progress is updated to reflect the number of confirmed files acquired versus the number expected. Once the MASTR-MS system has confirmed that run progress is at 100 percent, the run is marked complete and the run data is available to authorized users for download. Component files and Sample files are available for download separately, and Sample files can be packed into compressed archives (zip, tar.gz, tar.bzip) for efficient download, to minimize download sizes.\nMASTR-MS is designed in a generic form such that it accommodates the automatic capture and transfer of any type of data from an acquisition computer to the server. This feature allows MASTR-MS to be used with instruments from different vendors with different file types.\n\nDiscussion \nThe systematic tracking, analysis and sharing of complex datasets generated by high-throughput omics technologies such as those used in metabolomics represents a major and expanding challenge. Reliance on outdated methods for recording information about projects, experiments, samples and instruments is cumbersome and error-prone. The methodical management of lab data can be achieved by software solutions such as LIMS and electronic notebook systems. An ideal LIMS solution should be able to manage users and user privileges of the lab; manage the setting up of projects, experiments and samples; and manage the resulting data. It should be able to facilitate sharing of meta\/experimental data to other collaborating laboratories. The advantages of using task-specific LIMS over the old manual laboratory notebook or even simple spreadsheets are enormous. With well-designed systems such as LIMS solutions, search and retrieval becomes easy and efficient, especially in a lab that has been operating for several years, thereby having collected information on hundreds of projects and experiments. In addition, security plays an important role in LIMS solutions. Access to information and data about projects, experiments and samples would be controlled to be accessed only by authorized individuals. Finally, all information can be backed up to secure locations, thereby reducing the risk of accidental loss of data (Table 1).\n\n\n\n\n\n\n\nTable 1. User roles and access privileges\n\n\nUser type\n\nAccess privilege\n\n\nAdministrator\n\nComplete read-write access to all modules and nodes\n\n\nNode representative\n\nFor their specific node, complete read-write access to all modules\n\n\nProject manager\n\nFor their specific node, read-write access to only projects and experiments associated with them\n\n\nLab assistant\n\nFor their specific node, read-write access to only experiments associated with them\n\n\nClient\n\nRead access to only experiments associated with them\n\n\n\nMASTR-MS is a comprehensive web-based LIMS solution that has been tailor-made for metabolomic experiments and is suitable for implementation within a single laboratory environment or across a multi-node research consortium\/core facility. It (a) captures the entire lifecycle of a sample, from project and experimental design to the automatic capture and methodical storage of raw data generated by the multiple analytical instruments; (b) stores metadata about projects, experiments and samples and links the raw data with the metadata; (c) acts as a comprehensive electronic workbook; (d) acts as a storage solution for the vast amount of high throughput data generated by metabolomic experiments and (e) facilitates collaboration between different laboratories.\n\nScope of MASTR-MS \nMASTR-MS efficiently manages the lifecycle of a sample, capturing information from client communication through to establishing projects, experiments, samples and continuing to automatic capture of raw data from the analytical instruments. MASTR-MS also stores processed data along with results of any statistical analysis and project reports. By design, MASTR-MS does not provide tools for data processing or statistical analysis, allowing researchers maximum flexibility for data processing and analysis, while allowing processed data to be imported and linked to raw data.\nAn important function of MASTR-MS is to act as an electronic laboratory notebook. To facilitate this, information is collected through free-flowing text fields. The advantage of this approach is that it allows the users to enter the same types of information that they would enter in their traditional lab notebook. The limitation behind this approach is that the entries are not controlled for ontologies, and therefore adopting to standards becomes challenging. Changing the free text entry to controlled vocabulary and incorporating the current MSI standards, as well as adopting the metabolomics community standards (ISA-Tab, mw-Tab) will be considered in future iterations of MASTR-MS.\n\nComparison to similar software \nMASTR-MS offers a number of features that distinguish it from other metabolomics LIMS systems such as SetupX and Sesame. SetupX[19] is a web-based metabolomics LIMS solution that is XML compatible and built around a relational database management core. It is particularly oriented towards the capture and display of GC\u2013MS metabolomic data through its metabolic annotation database, BinBase.[20] SetupX is able to handle a wide variety of BioSources (spatial, historical, environmental and genotypic descriptions of biological objects undergoing metabolomic investigations) and Treatments (experimental alterations that influence the metabolic states of BioSources). Compared to SetupX, MASTR-MS has not associated its input fields to ontologies, although it is intended that this will be incorporated into future versions of MASTR-MS as international standards are increasingly being adopted. Compared to SetupX, MASTR-MS offers the following advantages. It is able to cover multiple collaborating labs with a single deployment; lab-based users can generate the sequence list of samples to be run in the analytical instruments, thereby saving time and reducing the possibility of human errors; raw data generated by analyses is automatically captured by MASTR-MS; the user management system is extensive; and collaborators and clients are able to interact with the nodes using the Quote Management System.\nSesame[21] is also a web-based, platform-independent LIMS. It is based on Java CORBA, a commercial and open-source RDBMS, and was originally developed to facilitate NMR-based structural genomics studies.[22] The Sesame module for metabolomics is called \"Lamp.\" The Lamp module was originally designed to process NMR metabolomic analyses of Arabidopsis, although it is flexible enough to be easily adapted to other biological systems and other analytical methods. It consists of a number of different \"Views\" which provide details about the data, the instruments, and system resources used in a given study. In Sesame, the Views are designed to operate on various kinds of data, and facilitate data capture, editing, processing, analysis, retrieval and report generation. Sesame is a broad LIMS solution whose origins are in structural and functional proteomics, managing data from NMR platforms. Lamp, the module of Sesame that manages metabolomics data, is one of nine application modules of Sesame and was originally designed to manage information about the expression and purification of proteins and store this information. As Sesame and Lamp were not originally designed for metabolomics, its functions and features do not directly reflect the workflow of a typical metabolomics experiment. For example, even though Sesame has an extensive user management system, it does not have the functionalities of MASTR-MS that was specifically designed for metabolomics, such as an exhaustive project, experiment and sample management system, the ability of users of the lab to generate the sequence list of samples to be run in the analytical instruments, automatic capture of raw data from instruments and the ability of collaborators and clients to interact with the nodes using the Quote Management System.\nIn addition to the above discussed open source solutions, there are several commercial LIMS solutions such as MetaboLIMS from Core Informatics[23], MetLIMS from BioCrates[24] and Clarity LIMS from GenoLogics.[25] Due to their commercial nature, their functions and features are not readily available to compare against MASTR-MS.\n\nConclusion \nThis paper describes MASTR-MS, a new, fully integrated, open-source LIMS solution specifically designed for metabolomics laboratories. MASTR-MS can be used to track and share metabolomics experiments within a single laboratory or across large collaborative networks. Its comprehensive functions and features enable researchers and facilities to effectively manage a wide range of different project and experimental data types, and it facilitate the mining of new and existing datasets. The generic design of the data management component of MASTR-MS ensures that it can be used with instruments from different vendors. In addition, we have found that MASTR-MS can provide a LIMS solution for other data-rich technology platforms, such as proteomics, NMR and imaging facilities. MASTR-MS already has considerable community support, and new features will continuously be incorporated, including the capacity for researchers to directly upload their metadata and data to public metabolomics repositories such as MetaboLights and the Metabolomics Workbench. In addition, a reporting and export function is being developed at the user level, enabling the user to query the system and download data. In order to make automatic querying and retrieval easy, an API for MASTR-MS is being planned as well.\n\nAvailability and requirements \nProject name: MASTR-MS\nProject home page: https:\/\/muccg.github.io\/mastr-ms\/\nOperating system(s): Server Installation: Centos 6.x (x86_64); Client: Any operating system and modern web browser can be used as the web client to access MASTR-MS; DataSync Client: Linux or Windows\nProgramming language: Python 2.7\nSoftware requirements: Apache 2.2 or higher, PostgreSQL 8.4 or higher\nLicense: GNU GPL v3\nAny restrictions to use by non-academics: See GNU GPL v3\n\nAcknowledgements \nThis project is supported by Bioplatforms Australia Ltd., the Australian National Collaborative Research Infrastructure Strategy Program and the Education Investment Fund Super Science Initiative. The authors gratefully acknowledge additional funding from the Australian National Health and Medical Research Council (APP634485, APP1055319) and the EU FP7 Project (HEALTH.2012.2.1.1-1-C): RD Connect: An integrated platform connecting databases, registries, biobanks and clinical bioinformatics for rare disease research. MJM is a NHMRC Principal Research Fellow. AB acknowledges the support of the ARC Centre of Excellence in Plant Cell Walls. The authors acknowledge the many contributions made by other researchers in the Bioplatforms Australia network, including Michael Clarke, Hayden Walker, Dorothee Hayne, Robert Trengove and Catherine Rawlinson.\nAdam Hunter, Saravanan Dayalan and David De Souza have contributed equally to this work.\n\nCompliance with ethical standards \nConflict of interest \nAll authors declare that they have no conflicts of interest.\n\nEthical approval \nThis article does not contain any studies with human participants or animals performed by any of the authors.\n\nSupplementary material \nSupplementary material 1 (S9A) (JPG 137 KB)\nSupplementary material 2 (S4) (PNG 48 KB)\nSupplementary material 3 (S5A\u2013D) (PNG 6836 KB)\nSupplementary material 4 (S7A\u2013B) (PNG 3593 KB)\nSupplementary material 5 (S8A\u2013B) (PNG 2913 KB)\n\nReferences \n\n\n\u2191 \"Metabolomics Australia\". Metabolomics Australia. http:\/\/www.metabolomics.net.au\/ . Retrieved 05 December 2014 .   \n\n\u2191 \"The Metabolomics Inovation Centre\". University of Alberta. http:\/\/www.metabolomicscentre.ca\/ . Retrieved 05 December 2014 .   \n\n\u2191 \"Metabolomics\". The Common Fund. National Institutes of Health, Office of Strategic Coordination. http:\/\/commonfund.nih.gov\/metabolomics\/index . Retrieved 05 December 2014 .   \n\n\u2191 \"MetaboHUB\". MetaboHUB Centre INRA Bordeaux - Aquitaine. http:\/\/www.metabohub.fr\/ . Retrieved 05 December 2014 .   \n\n\u2191 5.0 5.1 \"Guidelines on FAIR Data Management in Horizon 2020 - Version 3.0\" (PDF). European Commission. 26 July 2016. http:\/\/ec.europa.eu\/research\/participants\/data\/ref\/h2020\/grants_manual\/hi\/oa_pilot\/h2020-hi-oa-data-mgt_en.pdf . Retrieved 05 December 2014 .   \n\n\u2191 \"NIH Data Sharing Policy and Implementation Guidance\". Grants and Funding. National Institutes of Health. 05 March 2003. https:\/\/grants.nih.gov\/grants\/policy\/data_sharing\/data_sharing_guidance.htm . Retrieved 05 December 2014 .   \n\n\u2191 \"Guidance for researchers: Developing a data management and sharing plan\". Policy and position statements. Wellcome Trust. 2014. Archived from the original on 18 October 2014. https:\/\/web-beta.archive.org\/web\/20141018165611\/http:\/\/www.wellcome.ac.uk\/About-us\/Policy\/Spotlight-issues\/Data-sharing\/Guidance-for-researchers\/index.htm . Retrieved 05 December 2014 .   \n\n\u2191 \"Australian Code for the Responsible Conduct of Research\". National Health and Medical Research Council, Australia. 2007. https:\/\/www.nhmrc.gov.au\/guidelines-publications\/r39 . Retrieved 05 December 2014 .   \n\n\u2191 \"Data Policies\". Nature. Macmillan Publishers Limited. https:\/\/www.nature.com\/sdata\/policies\/data-policies . Retrieved 05 December 2014 .   \n\n\u2191 \"Instructions for authors: Research Articles\". GigaScience. BioMed Central Ltd. 2014. Archived from the original on 15 May 2014. https:\/\/web.archive.org\/web\/20140515012736\/http:\/\/www.gigasciencejournal.com:80\/authors\/instructions\/research . Retrieved 05 December 2014 .   \n\n\u2191 Haug, K.; Salek, R.M.; Conesa, P. (2013). \"MetaboLights--An open-access general-purpose repository for metabolomics studies and associated meta-data\". Nucleic Acids Research 41 (D1): D781-6. doi:10.1093\/nar\/gks1004. PMC PMC3531110. PMID 23109552. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531110 .   \n\n\u2191 \"Metabolics Workbench\". University of California San Diego. http:\/\/www.metabolomicsworkbench.org\/ . Retrieved 05 December 2014 .   \n\n\u2191 \"Python\". Python Software Foundation. https:\/\/www.python.org\/ . Retrieved 05 December 2014 .   \n\n\u2191 \"Django\". Django Software Foundation. https:\/\/www.djangoproject.com\/ . Retrieved 05 December 2014 .   \n\n\u2191 \"PostgreSQL\". PostgreSQL Global Development Group. https:\/\/www.postgresql.org\/ . Retrieved 05 December 2014 .   \n\n\u2191 \"MySQL\". Oracle Corporation. https:\/\/www.mysql.com\/ . Retrieved 05 December 2014 .   \n\n\u2191 \"wxWidgets\". wxWidgets Development Team. https:\/\/www.wxwidgets.org\/ . Retrieved 10 November 2016 .   \n\n\u2191 \"rsync\". Wayne Davison. https:\/\/rsync.samba.org\/ . Retrieved 05 December 2014 .   \n\n\u2191 Scholz, M.; Fiehn, O. (2007). \"SetupX -- A public study design database for metabolomic projects\". Pacific Symposium on Biocomputing 2007: 169\u201380. PMID 17990490.   \n\n\u2191 Skogerson, K.; Wohlgemuth, G.; Barupal, D.K.; Fiehn, O. (2011). \"The volatile compound BinBase mass spectral database\". BMC Bioinformatics 12: 321. doi:10.1186\/1471-2105-12-321. PMC PMC3199763. PMID 21816034. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3199763 .   \n\n\u2191 Zolnai, Zsolt; Lee, Peter T.; Li, Jing; Chapman, Michael R.; Newman, Craig S.; Phillips Jr., George N.; Rayment, Ivan; Ulrich, Eldon L.; Volkman, Brian F.; Markley, John L. (January 2003). \"Project management system for structural and functional proteomics: Sesame\" (PDF). Journal of Structural and Functional Genomics 4 (1): 11\u201323. doi:10.1023\/A:1024684404761. http:\/\/www.springerlink.com\/content\/p3u654x38832uv73\/fulltext.pdf .   \n\n\u2191 Markley, J.L.; Anderson, M.E.; Cui, Q. et al. (2007). \"New bioinformatics resources for metabolomics\". Pacific Symposium on Biocomputing 2007: 157\u2013168. PMID 17990489.   \n\n\u2191 \"Core Informatics\". Core Informatics, LLC. https:\/\/www.coreinformatics.com\/ . Retrieved October 2016 .   \n\n\u2191 \"Biocrates Life Sciences\". Biocrates Life Sciences AG. http:\/\/www.biocrates.com\/ . Retrieved October 2016 .   \n\n\u2191 \"Clarity LIMS Gold\". GenoLogics Life Sciences Software Inc.. https:\/\/www.genologics.com\/editions\/clarity-lims-gold\/ . Retrieved October 2016 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article's references were in alphabetical order; the references here are shown in order of appearance in the article due to the way the wiki processes references. In one case the original URL had changed; an archived version of the URL was used instead. In two other cases, the original authors inadvertently used an incorrect reference (for Sesame LIMS and for Java CORBA); a correct reference was added to each for this version. The supplemental figures mentioned in the text have a different name than the ones supplied at the end; a best-effort attempt has been made to match the supplemental figure numbers in the text to the files linked in the \"Supplementary material\" section. (Though S9B and S9C are mentioned, they don't seem to be included, unless they are referencing S9A again.) Some grammar and spelling corrections were also made.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\">https:\/\/www.limswiki.org\/index.php\/Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 7 July 2017, at 18:18.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,744 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","c1c21b43f0762acc3ad8147824a0e651_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_MASTR-MS_A_web-based_collaborative_laboratory_information_management_system_LIMS_for_metabolomics skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:MASTR-MS: A web-based collaborative laboratory information management system (LIMS) for metabolomics<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h3>\n<p>An increasing number of research <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratories<\/a> and core analytical facilities around the world are developing high throughput metabolomic analytical and data processing pipelines that are capable of handling hundreds to thousands of individual samples per year, often over multiple projects, collaborations and sample types. At present, there are no <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> (LIMS) that are specifically tailored for metabolomics laboratories that are capable of tracking samples and associated metadata from the beginning to the end of an experiment, including data processing and archiving, and which are also suitable for use in large institutional core facilities or multi-laboratory consortia as well as single laboratory environments.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h3>\n<p>Here we present <a href=\"https:\/\/www.limswiki.org\/index.php\/MASTR-MS\" title=\"MASTR-MS\" target=\"_blank\" class=\"wiki-link\" data-key=\"8e97b3a6750e0d0edfeb14613720dd24\">MASTR-MS<\/a>, a downloadable and installable LIMS solution that can be deployed either within a single laboratory or used to link workflows across a multisite network. It comprises the Node Management System that can be used to link and manage projects across one or multiple collaborating laboratories; the User Management System which defines different user groups and privileges of users; the Quote Management System where client quotes are managed; the Project Management System in which metadata is stored and all aspects of project management, including experimental setup, sample tracking and instrument analysis, are defined; and the Data Management System that allows the automatic capture and storage of raw and processed data from the analytical instruments to the LIMS.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h3>\n<p>MASTR-MS is a comprehensive LIMS solution specifically designed for metabolomics. It captures the entire lifecycle of a sample, starting from project and experiment design to sample analysis, data capture and storage. It acts as an <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_laboratory_notebook\" title=\"Electronic laboratory notebook\" target=\"_blank\" class=\"wiki-link\" data-key=\"a9fbbd5e0807980106763fab31f1e72f\">electronic notebook<\/a>, facilitating project management within a single laboratory or a multi-node collaborative environment. This software is being developed in close consultation with members of the metabolomics research community. It is freely available under the GNU GPL v3 license and can be accessed from <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/muccg.github.io\/mastr-ms\/\" target=\"_blank\">https:\/\/muccg.github.io\/mastr-ms\/<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Keywords\">Keywords<\/span><\/h3>\n<p>MASTR-MS, metabolomics, LIMS, omics \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Metabolomic approaches aim to detect and quantitate levels of all small molecules in a biological system and, together with other \"omic\" approaches, can be used to generate a systems-wide understanding of biological processes. Metabolomic approaches typically involve the use of advanced mass spectrometry and nuclear magnetic resonance (NMR) spectrometry platforms to maximize coverage of the chemically diverse metabolites that make up biological systems. In many cases, these analytical platforms are located in institutional and\/or national core facilities that offer a range of metabolomics capabilities to researchers.<sup id=\"rdp-ebb-cite_ref-MetabolomicsAust_1-0\" class=\"reference\"><a href=\"#cite_note-MetabolomicsAust-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MetabolomicsInnovCent_2-0\" class=\"reference\"><a href=\"#cite_note-MetabolomicsInnovCent-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NIHMetabolomics_3-0\" class=\"reference\"><a href=\"#cite_note-NIHMetabolomics-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MetaboHUB_4-0\" class=\"reference\"><a href=\"#cite_note-MetaboHUB-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ECGuidelines16_5-0\" class=\"reference\"><a href=\"#cite_note-ECGuidelines16-5\" rel=\"external_link\">[5]<\/a><\/sup> These core facilities, as well as individual research groups with sophisticated metabolomics infrastructure and capability are faced with the challenge of tracking large numbers of samples and the associated metadata, and linking this information with the raw datasets generated by multiple analytical platforms, as well as processed down-stream data sets. Data handling extends beyond collection and curation of raw data, to the management of metadata that defines how the raw data is generated. Major funding agencies, such as Europe\u2019s Horizon 2020<sup id=\"rdp-ebb-cite_ref-ECGuidelines16_5-1\" class=\"reference\"><a href=\"#cite_note-ECGuidelines16-5\" rel=\"external_link\">[5]<\/a><\/sup>, the NIH<sup id=\"rdp-ebb-cite_ref-NIHDataSharing03_6-0\" class=\"reference\"><a href=\"#cite_note-NIHDataSharing03-6\" rel=\"external_link\">[6]<\/a><\/sup>, The Wellcome Trust<sup id=\"rdp-ebb-cite_ref-WTGuidance14_7-0\" class=\"reference\"><a href=\"#cite_note-WTGuidance14-7\" rel=\"external_link\">[7]<\/a><\/sup> and Australia\u2019s NHMRC<sup id=\"rdp-ebb-cite_ref-AGNHMRC_AustralianCode07_8-0\" class=\"reference\"><a href=\"#cite_note-AGNHMRC_AustralianCode07-8\" rel=\"external_link\">[8]<\/a><\/sup> have established data management plans that researchers are expected to follow in order to capture, store and share data generated by their grants. Scientific journals are also increasingly requesting that experimental data and metadata associated with metabolomics experiments are made available to the scientific community<sup id=\"rdp-ebb-cite_ref-NatureDPs_9-0\" class=\"reference\"><a href=\"#cite_note-NatureDPs-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GigaScienceResearch_10-0\" class=\"reference\"><a href=\"#cite_note-GigaScienceResearch-10\" rel=\"external_link\">[10]<\/a><\/sup>, leading to the establishment of data repositories, such as MetaboLights<sup id=\"rdp-ebb-cite_ref-HaugMetabo13_11-0\" class=\"reference\"><a href=\"#cite_note-HaugMetabo13-11\" rel=\"external_link\">[11]<\/a><\/sup> and Metabolomics Workbench.<sup id=\"rdp-ebb-cite_ref-UCSDMetab_12-0\" class=\"reference\"><a href=\"#cite_note-UCSDMetab-12\" rel=\"external_link\">[12]<\/a><\/sup>\n<\/p><p>LIMS are software solutions that aim to manage the entire workflow of a laboratory. A number of LIMS have been developed or adapted from other applications for curating metabolomics experiments and data management (e.g., SetupX, <a href=\"https:\/\/www.limswiki.org\/index.php\/Sesame_LIMS\" title=\"Sesame LIMS\" target=\"_blank\" class=\"wiki-link\" data-key=\"e834d2f189dbbb034ac9aaace88e5190\">Sesame<\/a>). While these LIMS have features that allow capture of project metadata, experiments and samples, data storage, and data sharing, they exhibit a number of limitations around their capacity to accommodate different vendor instruments and have restricted functionalities to facilitate a collaborative configuration between geographically distributed laboratories. In this paper we present MASTR-MS, the first wholly functional, open-source LIMS solution specifically designed for metabolomics laboratories.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<p>MASTR-MS runs as a Python<sup id=\"rdp-ebb-cite_ref-Python_13-0\" class=\"reference\"><a href=\"#cite_note-Python-13\" rel=\"external_link\">[13]<\/a><\/sup> web application built on the Django<sup id=\"rdp-ebb-cite_ref-Django_14-0\" class=\"reference\"><a href=\"#cite_note-Django-14\" rel=\"external_link\">[14]<\/a><\/sup> framework, utilising a <a href=\"https:\/\/www.limswiki.org\/index.php\/PostgreSQL\" title=\"PostgreSQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"a5dd945cdcb63e2d8f7a5edb3a896d82\">PostgreSQL<\/a><sup id=\"rdp-ebb-cite_ref-PostgreSQL_15-0\" class=\"reference\"><a href=\"#cite_note-PostgreSQL-15\" rel=\"external_link\">[15]<\/a><\/sup> or <a href=\"https:\/\/www.limswiki.org\/index.php\/MySQL\" title=\"MySQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"35005451bfcd508bce47c58e72260128\">MySQL<\/a><sup id=\"rdp-ebb-cite_ref-MySQL_16-0\" class=\"reference\"><a href=\"#cite_note-MySQL-16\" rel=\"external_link\">[16]<\/a><\/sup> relational database. MASTR-MS leverages the functionality of the Django framework for user management, users permissions and security. Django is a mature web framework and provides multiple security tools and mechanisms. For example, specific protection is provided against cross-site scripting (XSS), cross-site request forgery (CSRF), SQL injection and clickjacking. A security middleware is also used to enforce SSL\/HTTPS for all traffic. MASTR-MS is built using open-source components and communicates using open standards. The client side browser interface leverages Javascript and AJAX for fluid data display and submission, giving a user experience much like a desktop application, but with the flexibility of being available from any internet-connected location on any operating system, with no client-side download or installation.\n<\/p><p>The DataSync Client is a small desktop application that runs on an instrument\u2019s acquisition computer. This software constantly communicates with the MASTR-MS server and is responsible for transferring raw data from the acquisition computer to the MASTR-MS repository (Supplemental Fig. S9A). The DataSync Client is written in the Python programming language using the wxWidgets<sup id=\"rdp-ebb-cite_ref-wxWidgets_17-0\" class=\"reference\"><a href=\"#cite_note-wxWidgets-17\" rel=\"external_link\">[17]<\/a><\/sup> GUI library and runs on Windows and Linux systems. Data is uploaded using the rsync protocol<sup id=\"rdp-ebb-cite_ref-rsync_18-0\" class=\"reference\"><a href=\"#cite_note-rsync-18\" rel=\"external_link\">[18]<\/a><\/sup> and the libraries and plugins required for this are included in the installation package.\n<\/p><p>As the MASTR-MS server-side component is written in the Python 2.7 programming language, any operating system that has Python 2.7 available for running web applications with a web server can run the application. In practice, the application has only been tested on the Linux operating system and the Apache web server. For installation, operating system packages are available in RPM format for CentOS 6.5. Similarly, as the DataSync Client is also written in Python 2.7 it can run on any operating system that has Python 2.7 available. However it is typically installed on a Windows platform with a connected analytical instrument. For this reason, the DataSync Client is distributed as a Windows executable (.exe) installer. The DataSync Client application is also self-updating by means of a user option to upgrade to a newer version if available.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_2\">Results<\/span><\/h2>\n<p>MASTR-MS is a web-based LIMS solution for metabolomics laboratories. The different modules of MASTR-MS allow users to: \n<\/p>\n<ul><li> Track all metabolomics samples and associated meta-, analytical- and processed data sets. This starts from the capture of client\/collaborator communication; the establishment of new projects, experimental design and sample definitions; and the automatic capture of raw data generated by the instruments.<\/li>\n<li> Develop an electronic notebook, where users record all relevant information about projects and experiments in MASTR-MS, thus allowing multiple users to work on the same project.<\/li>\n<li> Methodically manage the vast amount of data generated by the analytical instruments, by associating it with the project, experiment and sample details.<\/li>\n<li> Facilitate collaboration between geographically distributed laboratories through the sharing of projects and experiment data.<\/li><\/ul>\n<p>MASTR-MS is equally suited for use in either a large core facility or single-\/multi-laboratory environment. Thus, both large national facilities and small individual laboratories would equally benefit from using MASTR-MS.\n<\/p><p>MASTR-MS comprises five major modules: (1) the Node Management System, (2) the User Management System, (3) the Quote Management System, (4) the Project Management System and (5) the Data Management System. Figure 1 shows the workflow of MASTR-MS using the different functionalities and features. These functions are described in detail below. The user is initially connected to the dashboard when they first log into MASTR-MS, and the available functions are tailored to the level of access of the user. The dashboard gives an \u201cat-a-glance\u201d summary of recent activity on the site and items requiring attention. Depending on the user\u2019s status\/level of access, the dashboard shows pending user requests, quotes requiring attention, recently created \/ modified projects, and recently created \/ modified experiments.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Hunter_Metabolomics2017_13-2.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"4823d9a2cc9bd9eb98116af09ccc10e3\"><img alt=\"Fig1 Hunter Metabolomics2017 13-2.gif\" src=\"https:\/\/www.limswiki.org\/images\/6\/6a\/Fig1_Hunter_Metabolomics2017_13-2.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Overview of MASTR-MS system workflow<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Node_management_system\">Node management system<\/span><\/h3>\n<p>This module allows the addition of multiple laboratories to be part of a single MASTR-MS network. For example, a group of geographically dispersed laboratories can have a single deployment of MASTR-MS and share projects and experiments. Such a setup would be established by the module through the generation of different nodes. On the other hand, MASTR-MS can be used within a single laboratory environment in which this module would comprise a single node.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"User_management_system\">User management system<\/span><\/h3>\n<p>This module defines the different user groups used in MASTR-MS. Each user group has different privileges and permissions to access the different functionalities of MASTR-MS. In addition, this module allows the generation and management of users of the system. MASTR-MS has several user groups.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Systems_administrator\">Systems administrator<\/span><\/h4>\n<p>This user group has access to all functionalities of MASTR-MS. There would normally be one assigned Systems Administrator who would act as the query point for all other users accessing the system, although it is possible to have more than one Systems Administrator. The Systems Administrator has a Laboratory Name assigned to their account (like all other users), allowing a nominated user, usually a member of the organization\/laboratory that is hosting the project to act as the Systems Administrator. The Systems Administrator can add new users to the system, assign user groups to any users in any laboratory, edit details of users and delete users of any laboratory.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Administrator\">Administrator<\/span><\/h4>\n<p>This user group has full access to all projects, experiments and experimental data, user accounts and quotes within MASTR-MS, regardless of node. This user group allows selected users to view all projects and experiments across different nodes, allowing seamless sharing and collaboration of data across nodes. Where multiple laboratories have a single MASTR-MS deployment, but prefer not to share projects and experiments, no users would be assigned the Administrator role.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Node_representative\">Node representative<\/span><\/h4>\n<p>This user group has full access to quotes for their node and are the preferred contact for quotes and projects run by this node (detailed more in the \"Quote management system\" section). In a multi-node setup there would typically be at least one user assigned to this group per node.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Project_leader\">Project leader<\/span><\/h4>\n<p>This user group is able to create new projects and experiments for their node. Additionally, this group is able to assign staff to specific projects and experiments.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Staff\">Staff<\/span><\/h4>\n<p>Users of this group are able to participate in the projects and experiments for their node.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Client\">Client<\/span><\/h4>\n<p>All other users of the system are clients. This group has no privileges other than viewing the progress of projects to which they have been assigned.\n<\/p><p>Any user of the system can update their own user record and change their password at any time.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Quote_management_system\">Quote management system<\/span><\/h3>\n<p>This module was designed specifically for core facilities that provide metabolomic services to client researchers. Potential clients can request a pricing quote for running samples of an experiment through the quote request system without having to sign up for an account. At a nominated stage, clients are required to register in MASTR-MS by completing a short information dialog box. This module allows collection of contact details and information about the nature of the request. Files in various formats can be attached to this module. In a multi-node facility, the user can either direct their quote to a specific node with relevant expertise or they can select \"Don\u2019t Know\" to have all the Node Representatives alerted.\n<\/p><p>Quote requests made by clients and collaborators that are made through the system are tracked and marked if they have not been attended to yet, so that Node Representatives can quickly see new quotes which require attention. Quotes can only be seen by members of the node to which they were sent, unless the \"Don\u2019t Know\" option was selected. Node Representatives are able to forward quotes to other nodes if required. The Node Representatives can then begin a dialogue with the potential client and with their team, clarifying the task, and providing formal quotes, attached as PDFs if necessary. Each step of the communications process is time-stamped and tracked within this module. The quote requests and any resulting quotes would eventually be associated with a project and experiment through a selection option in the Experimental Design stage. All documentation relating to the project, including the client and quote issued for the project, along with the project and experimental setup, is thus kept together.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Project_management_system\">Project management system<\/span><\/h3>\n<p>This module allows the management of projects, experiments, and samples as well as the creation of analytical sample runs. As detailed above, users of different user groups are able to create projects and experiments. When a project is created by either a MASTR-MS Administrator or Project Leader, it can be linked to a specific client from the user list. This allows the client to monitor how the project is progressing. Assigning a Project Manager to the project allows those users to manage all aspects of a project, experiment creation and further access control on an experiment-by-experiment basis (Supplemental Fig. S4). As sample metadata is linked to all experiments within MASTR-MS, sample classes and\/or individual samples can be organized into groups and subsequently analyzed on an instrument.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Experiment_details\">Experiment details<\/span><\/h4>\n<p>The Experiment Status defaults to \"New\" when first opened, and all experiment metadata is captured in this field (Supplemental Fig. S5A). Once the experiment design has been completed, the Project Manager can change the setting to \"Designed\" to prevent further changes. The experiment can also be linked to a formal quote that has been previously entered in the quotes system, and if needed, can be assigned an internal job number.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Access_control.2Froles\">Access control\/roles<\/span><\/h4>\n<p>Users can be assigned to an experiment, giving them access to edit the experimental workflow and create samples and runs. Client users can also be added here, giving them access to project progress information (Supplemental Fig. S5B).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Sample_metadata\">Sample metadata<\/span><\/h4>\n<p>MASTR-MS uses sample metadata in order to generate sample classes, which can then be populated with individual samples (Supplemental Fig. S5C).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Origin.2Forgans.2Fparts_metadata\">Origin\/organs\/parts metadata<\/span><\/h4>\n<p>The first metadata category is the Origin field, which contains information on sample origin and preparation (Supplemental Fig. S5D). Different metadata fields are available depending on whether the source is Microbial, Plant, Animal, Human, Synthetic, or Other.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Timeline.2Ftreatment_metadata\">Timeline\/treatment metadata<\/span><\/h4>\n<p>MASTR-MS also accepts time course and treatment metadata, where samples have been collected over multiple time points, or after different experimental treatments. The Origin, Timeline, and Treatment fields are then used to automatically generate sample classes.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Sample_preparation\">Sample preparation<\/span><\/h4>\n<p>MASTR-MS allows an upload of a standard operating procedure (SOP) document to be associated with an experiment. Multiple SOPs can be uploaded and additional notes recorded for each. A SOP is linked with methods used during runs at the time of setting up a run. The SOP is linked at the experiment level, and the option of choosing methods is provided under the runs level. This is to incorporate the option where a user would like to run multiple methods during a run (either by resampling the same vial or from a different vial).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Automatic_sample_class_generation\">Automatic sample class generation<\/span><\/h4>\n<p>Based on the metadata entered in the Origin, Timeline, and Treatment steps, sample classes are automatically generated based on permutations of the available metadata (Supplemental Fig. S7A). If abbreviations have been provided for a particular metadata category, these will be used during sample class generation. Samples can then be created in each sample class.\n<\/p><p>Samples can then be viewed and collected together to form a run on a designated analytical instrument platform (Supplemental Fig. S7B). Additional sample information can be imported via CSV and exported from MASTR-MS in the same way. Samples can be randomized before putting them into a run if desired.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Runs\">Runs<\/span><\/h4>\n<p>Selected samples are added to a new or existing run by clicking the \"Add Selected Samples to Run\" button. This will display a dialog allowing the user to add either the samples to a new run or to any previous run which is still unlocked for editing (Supplemental Fig. S8A). Runs continue to be unlocked as long as a worklist has not yet been generated for them. Locked runs can be edited and reused if needed using the \u201cRun Cloning\u201d feature, which will duplicate the run data into a new unlocked run.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Worklist_generation\">Worklist generation<\/span><\/h4>\n<p>The goal of run configuration is to streamline sample analysis and generate instrument worklists in a convenient and flexible manner. After sample data has been added to a run, the order and sequencing of additional run elements (Sweeps, Solvents, etc.) can be added via the Rules Generator.\n<\/p><p>The Rules Generator provides a customizable set of steps (rules) which dictate how worklists are built. It consists of a Start Block, Sample Block, and End Block, each of which allows the insertion of non-sample components into the worklist. These include Pooled Biological QC, Sweep, Reagent Blank, Solvent Blank and Pure Standard.\n<\/p><p>The sample block, containing the experiment samples, allows <i>n<\/i> components to be inserted every <i>m<\/i> samples, in random or position order (Supplemental Fig. S8B). Once all three blocks have been designed, the Rule Generator can be enabled, disabling further editing and making the rule available for inclusion in run worklist generation. Rule Generators can be restricted to use by a single user, an entire node, or everybody on the system. Enabled Rule Generators can be cloned in order to generate a new version, which can then be extended and modified.\n<\/p><p>To generate a worklist within a run, the user selects an instrument (configured and made available by Administrators) and a Rule Generator, if needed, and clicks the \"Generate Worklist\" button. Once the worklist is generated, further modification of the run is not possible. The specific worklist format is customizable by site administrators to provide flexibility among various instrument models. Once the worklist is generated, it can be used with the instrument to automate the raw data collection process.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_management_system\">Data management system<\/span><\/h3>\n<p>This module facilitates the capture and storage of raw data produced by the instruments. The raw data is captured by the DataSync Client as detailed below and is linked to associated project and experiment details. In addition, post-processed data and any other related files such as presentations, reports and papers can be linked to the data.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Data_acquisition_and_the_DataSync_Client\">Data acquisition and the DataSync Client<\/span><\/h4>\n<p>The DataSync Client allows data to be transferred from connected instruments at nominated frequencies and will run in the background of the acquisition computers as an icon in the System Tray. The software is fully integrated with the MASTR-MS web application. When data synchronization is requested, either scheduled or manually, the DataSync Client communicates with the MASTR-MS system to query all incomplete experiment runs which have been configured for the connected instrument. It then searches the acquired data for required files and transmits them to the MASTR-MS repository via a configurable rsync transport, allowing compression and check-summing for efficient data transfer. The configuration options for individual DataSync Nodes are fully configurable via the MASTR-MS administration interface.\n<\/p><p>To enable DataSync Client uploads on the instrument, the user simply selects the connected instrument from the list which has been configured on the MASTR-MS system and enters the Rsync username which they have been assigned (Supplemental Fig. S9B). OpenSSH Public Keys can be uploaded to the MASTR-MS system for secure password-less usage, which allows the client to run seamless automated data uploads without need for operator intervention.\n<\/p><p>The DataSync Client can also be configured with some advanced options. Data archival allows the raw sample data to be automatically replicated in a specified location (e.g., on another hard disk) once confirmation of upload has been achieved, allowing the original data to be deleted if desired.\n<\/p><p>The software can also be forced to re-synchronize experiment data that has been marked as complete in case the need arises (Supplemental Fig. S9C).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Run_progress\">Run progress<\/span><\/h4>\n<p>As data is synced with the MASTR-MS system, run progress is updated to reflect the number of confirmed files acquired versus the number expected. Once the MASTR-MS system has confirmed that run progress is at 100 percent, the run is marked complete and the run data is available to authorized users for download. Component files and Sample files are available for download separately, and Sample files can be packed into compressed archives (zip, tar.gz, tar.bzip) for efficient download, to minimize download sizes.\n<\/p><p>MASTR-MS is designed in a generic form such that it accommodates the automatic capture and transfer of any type of data from an acquisition computer to the server. This feature allows MASTR-MS to be used with instruments from different vendors with different file types.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>The systematic tracking, analysis and sharing of complex datasets generated by high-throughput omics technologies such as those used in metabolomics represents a major and expanding challenge. Reliance on outdated methods for recording information about projects, experiments, samples and instruments is cumbersome and error-prone. The methodical management of lab data can be achieved by software solutions such as LIMS and electronic notebook systems. An ideal LIMS solution should be able to manage users and user privileges of the lab; manage the setting up of projects, experiments and samples; and manage the resulting data. It should be able to facilitate sharing of meta\/experimental data to other collaborating laboratories. The advantages of using task-specific LIMS over the old manual <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_notebook\" title=\"Laboratory notebook\" target=\"_blank\" class=\"wiki-link\" data-key=\"be60c7be96aba8e9a84537fd8835fa54\">laboratory notebook<\/a> or even simple spreadsheets are enormous. With well-designed systems such as LIMS solutions, search and retrieval becomes easy and efficient, especially in a lab that has been operating for several years, thereby having collected information on hundreds of projects and experiments. In addition, security plays an important role in LIMS solutions. Access to information and data about projects, experiments and samples would be controlled to be accessed only by authorized individuals. Finally, all information can be backed up to secure locations, thereby reducing the risk of accidental loss of data (Table 1).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> User roles and access privileges\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">User type\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Access privilege\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Administrator\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Complete read-write access to all modules and nodes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Node representative\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">For their specific node, complete read-write access to all modules\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Project manager\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">For their specific node, read-write access to only projects and experiments associated with them\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lab assistant\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">For their specific node, read-write access to only experiments associated with them\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Client\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Read access to only experiments associated with them\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>MASTR-MS is a comprehensive web-based LIMS solution that has been tailor-made for metabolomic experiments and is suitable for implementation within a single laboratory environment or across a multi-node research consortium\/core facility. It (a) captures the entire lifecycle of a sample, from project and experimental design to the automatic capture and methodical storage of raw data generated by the multiple analytical instruments; (b) stores metadata about projects, experiments and samples and links the raw data with the metadata; (c) acts as a comprehensive electronic workbook; (d) acts as a storage solution for the vast amount of high throughput data generated by metabolomic experiments and (e) facilitates collaboration between different laboratories.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Scope_of_MASTR-MS\">Scope of MASTR-MS<\/span><\/h3>\n<p>MASTR-MS efficiently manages the lifecycle of a sample, capturing information from client communication through to establishing projects, experiments, samples and continuing to automatic capture of raw data from the analytical instruments. MASTR-MS also stores processed data along with results of any statistical analysis and project reports. By design, MASTR-MS does not provide tools for data processing or statistical analysis, allowing researchers maximum flexibility for data processing and analysis, while allowing processed data to be imported and linked to raw data.\n<\/p><p>An important function of MASTR-MS is to act as an electronic laboratory notebook. To facilitate this, information is collected through free-flowing text fields. The advantage of this approach is that it allows the users to enter the same types of information that they would enter in their traditional lab notebook. The limitation behind this approach is that the entries are not controlled for ontologies, and therefore adopting to standards becomes challenging. Changing the free text entry to controlled vocabulary and incorporating the current MSI standards, as well as adopting the metabolomics community standards (ISA-Tab, mw-Tab) will be considered in future iterations of MASTR-MS.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Comparison_to_similar_software\">Comparison to similar software<\/span><\/h3>\n<p>MASTR-MS offers a number of features that distinguish it from other metabolomics LIMS systems such as SetupX and Sesame. SetupX<sup id=\"rdp-ebb-cite_ref-ScholzSetupX07_19-0\" class=\"reference\"><a href=\"#cite_note-ScholzSetupX07-19\" rel=\"external_link\">[19]<\/a><\/sup> is a web-based metabolomics LIMS solution that is XML compatible and built around a relational database management core. It is particularly oriented towards the capture and display of GC\u2013MS metabolomic data through its metabolic annotation database, BinBase.<sup id=\"rdp-ebb-cite_ref-SkogersonTheVol11_20-0\" class=\"reference\"><a href=\"#cite_note-SkogersonTheVol11-20\" rel=\"external_link\">[20]<\/a><\/sup> SetupX is able to handle a wide variety of BioSources (spatial, historical, environmental and genotypic descriptions of biological objects undergoing metabolomic investigations) and Treatments (experimental alterations that influence the metabolic states of BioSources). Compared to SetupX, MASTR-MS has not associated its input fields to ontologies, although it is intended that this will be incorporated into future versions of MASTR-MS as international standards are increasingly being adopted. Compared to SetupX, MASTR-MS offers the following advantages. It is able to cover multiple collaborating labs with a single deployment; lab-based users can generate the sequence list of samples to be run in the analytical instruments, thereby saving time and reducing the possibility of human errors; raw data generated by analyses is automatically captured by MASTR-MS; the user management system is extensive; and collaborators and clients are able to interact with the nodes using the Quote Management System.\n<\/p><p>Sesame<sup id=\"rdp-ebb-cite_ref-ZolnaiProject03_21-0\" class=\"reference\"><a href=\"#cite_note-ZolnaiProject03-21\" rel=\"external_link\">[21]<\/a><\/sup> is also a web-based, platform-independent LIMS. It is based on Java CORBA, a commercial and open-source RDBMS, and was originally developed to facilitate NMR-based structural genomics studies.<sup id=\"rdp-ebb-cite_ref-Markley_22-0\" class=\"reference\"><a href=\"#cite_note-Markley-22\" rel=\"external_link\">[22]<\/a><\/sup> The Sesame module for metabolomics is called \"Lamp.\" The Lamp module was originally designed to process NMR metabolomic analyses of <i>Arabidopsis<\/i>, although it is flexible enough to be easily adapted to other biological systems and other analytical methods. It consists of a number of different \"Views\" which provide details about the data, the instruments, and system resources used in a given study. In Sesame, the Views are designed to operate on various kinds of data, and facilitate data capture, editing, processing, analysis, retrieval and report generation. Sesame is a broad LIMS solution whose origins are in structural and functional proteomics, managing data from NMR platforms. Lamp, the module of Sesame that manages metabolomics data, is one of nine application modules of Sesame and was originally designed to manage information about the expression and purification of proteins and store this information. As Sesame and Lamp were not originally designed for metabolomics, its functions and features do not directly reflect the workflow of a typical metabolomics experiment. For example, even though Sesame has an extensive user management system, it does not have the functionalities of MASTR-MS that was specifically designed for metabolomics, such as an exhaustive project, experiment and sample management system, the ability of users of the lab to generate the sequence list of samples to be run in the analytical instruments, automatic capture of raw data from instruments and the ability of collaborators and clients to interact with the nodes using the Quote Management System.\n<\/p><p>In addition to the above discussed open source solutions, there are several commercial LIMS solutions such as MetaboLIMS from Core Informatics<sup id=\"rdp-ebb-cite_ref-CoreInfor_23-0\" class=\"reference\"><a href=\"#cite_note-CoreInfor-23\" rel=\"external_link\">[23]<\/a><\/sup>, MetLIMS from BioCrates<sup id=\"rdp-ebb-cite_ref-BioCrates_24-0\" class=\"reference\"><a href=\"#cite_note-BioCrates-24\" rel=\"external_link\">[24]<\/a><\/sup> and Clarity LIMS from GenoLogics.<sup id=\"rdp-ebb-cite_ref-ClarityLIMS_25-0\" class=\"reference\"><a href=\"#cite_note-ClarityLIMS-25\" rel=\"external_link\">[25]<\/a><\/sup> Due to their commercial nature, their functions and features are not readily available to compare against MASTR-MS.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion_2\">Conclusion<\/span><\/h2>\n<p>This paper describes MASTR-MS, a new, fully integrated, open-source LIMS solution specifically designed for metabolomics laboratories. MASTR-MS can be used to track and share metabolomics experiments within a single laboratory or across large collaborative networks. Its comprehensive functions and features enable researchers and facilities to effectively manage a wide range of different project and experimental data types, and it facilitate the mining of new and existing datasets. The generic design of the data management component of MASTR-MS ensures that it can be used with instruments from different vendors. In addition, we have found that MASTR-MS can provide a LIMS solution for other data-rich technology platforms, such as proteomics, NMR and imaging facilities. MASTR-MS already has considerable community support, and new features will continuously be incorporated, including the capacity for researchers to directly upload their metadata and data to public metabolomics repositories such as MetaboLights and the Metabolomics Workbench. In addition, a reporting and export function is being developed at the user level, enabling the user to query the system and download data. In order to make automatic querying and retrieval easy, an API for MASTR-MS is being planned as well.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Availability_and_requirements\">Availability and requirements<\/span><\/h2>\n<p><b>Project name<\/b>: MASTR-MS\n<\/p><p><b>Project home page<\/b>: <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/muccg.github.io\/mastr-ms\/\" target=\"_blank\">https:\/\/muccg.github.io\/mastr-ms\/<\/a>\n<\/p><p><b>Operating system(s)<\/b>: Server Installation: Centos 6.x (x86_64); Client: Any operating system and modern web browser can be used as the web client to access MASTR-MS; DataSync Client: Linux or Windows\n<\/p><p><b>Programming language<\/b>: Python 2.7\n<\/p><p><b>Software requirements<\/b>: Apache 2.2 or higher, PostgreSQL 8.4 or higher\n<\/p><p><b>License<\/b>: GNU GPL v3\n<\/p><p><b>Any restrictions to use by non-academics<\/b>: See GNU GPL v3\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This project is supported by Bioplatforms Australia Ltd., the Australian National Collaborative Research Infrastructure Strategy Program and the Education Investment Fund Super Science Initiative. The authors gratefully acknowledge additional funding from the Australian National Health and Medical Research Council (APP634485, APP1055319) and the EU FP7 Project (HEALTH.2012.2.1.1-1-C): RD Connect: An integrated platform connecting databases, registries, biobanks and clinical bioinformatics for rare disease research. MJM is a NHMRC Principal Research Fellow. AB acknowledges the support of the ARC Centre of Excellence in Plant Cell Walls. The authors acknowledge the many contributions made by other researchers in the Bioplatforms Australia network, including Michael Clarke, Hayden Walker, Dorothee Hayne, Robert Trengove and Catherine Rawlinson.\n<\/p><p>Adam Hunter, Saravanan Dayalan and David De Souza have contributed equally to this work.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Compliance_with_ethical_standards\">Compliance with ethical standards<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>All authors declare that they have no conflicts of interest.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Ethical_approval\">Ethical approval<\/span><\/h3>\n<p>This article does not contain any studies with human participants or animals performed by any of the authors.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_material\">Supplementary material<\/span><\/h2>\n<p><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1007%2Fs11306-016-1142-2\/MediaObjects\/11306_2016_1142_MOESM1_ESM.jpg\" target=\"_blank\">Supplementary material 1 (S9A)<\/a> (JPG 137 KB)\n<\/p><p><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1007%2Fs11306-016-1142-2\/MediaObjects\/11306_2016_1142_MOESM2_ESM.png\" target=\"_blank\">Supplementary material 2 (S4)<\/a> (PNG 48 KB)\n<\/p><p><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1007%2Fs11306-016-1142-2\/MediaObjects\/11306_2016_1142_MOESM3_ESM.png\" target=\"_blank\">Supplementary material 3 (S5A\u2013D)<\/a> (PNG 6836 KB)\n<\/p><p><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1007%2Fs11306-016-1142-2\/MediaObjects\/11306_2016_1142_MOESM4_ESM.png\" target=\"_blank\">Supplementary material 4 (S7A\u2013B)<\/a> (PNG 3593 KB)\n<\/p><p><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1007%2Fs11306-016-1142-2\/MediaObjects\/11306_2016_1142_MOESM5_ESM.png\" target=\"_blank\">Supplementary material 5 (S8A\u2013B)<\/a> (PNG 2913 KB)\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-MetabolomicsAust-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MetabolomicsAust_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.metabolomics.net.au\/\" target=\"_blank\">\"Metabolomics Australia\"<\/a>. Metabolomics Australia<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.metabolomics.net.au\/\" target=\"_blank\">http:\/\/www.metabolomics.net.au\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Metabolomics+Australia&rft.atitle=&rft.pub=Metabolomics+Australia&rft_id=http%3A%2F%2Fwww.metabolomics.net.au%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MetabolomicsInnovCent-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MetabolomicsInnovCent_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.metabolomicscentre.ca\/\" target=\"_blank\">\"The Metabolomics Inovation Centre\"<\/a>. University of Alberta<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.metabolomicscentre.ca\/\" target=\"_blank\">http:\/\/www.metabolomicscentre.ca\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Metabolomics+Inovation+Centre&rft.atitle=&rft.pub=University+of+Alberta&rft_id=http%3A%2F%2Fwww.metabolomicscentre.ca%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NIHMetabolomics-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NIHMetabolomics_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/commonfund.nih.gov\/metabolomics\/index\" target=\"_blank\">\"Metabolomics\"<\/a>. <i>The Common Fund<\/i>. National Institutes of Health, Office of Strategic Coordination<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/commonfund.nih.gov\/metabolomics\/index\" target=\"_blank\">http:\/\/commonfund.nih.gov\/metabolomics\/index<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Metabolomics&rft.atitle=The+Common+Fund&rft.pub=National+Institutes+of+Health%2C+Office+of+Strategic+Coordination&rft_id=http%3A%2F%2Fcommonfund.nih.gov%2Fmetabolomics%2Findex&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MetaboHUB-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MetaboHUB_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.metabohub.fr\/\" target=\"_blank\">\"MetaboHUB\"<\/a>. MetaboHUB Centre INRA Bordeaux - Aquitaine<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.metabohub.fr\/\" target=\"_blank\">http:\/\/www.metabohub.fr\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MetaboHUB&rft.atitle=&rft.pub=MetaboHUB+Centre+INRA+Bordeaux+-+Aquitaine&rft_id=http%3A%2F%2Fwww.metabohub.fr%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ECGuidelines16-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ECGuidelines16_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-ECGuidelines16_5-1\" rel=\"external_link\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ec.europa.eu\/research\/participants\/data\/ref\/h2020\/grants_manual\/hi\/oa_pilot\/h2020-hi-oa-data-mgt_en.pdf\" target=\"_blank\">\"Guidelines on FAIR Data Management in Horizon 2020 - Version 3.0\"<\/a> (PDF). European Commission. 26 July 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ec.europa.eu\/research\/participants\/data\/ref\/h2020\/grants_manual\/hi\/oa_pilot\/h2020-hi-oa-data-mgt_en.pdf\" target=\"_blank\">http:\/\/ec.europa.eu\/research\/participants\/data\/ref\/h2020\/grants_manual\/hi\/oa_pilot\/h2020-hi-oa-data-mgt_en.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Guidelines+on+FAIR+Data+Management+in+Horizon+2020+-+Version+3.0&rft.atitle=&rft.date=26+July+2016&rft.pub=European+Commission&rft_id=http%3A%2F%2Fec.europa.eu%2Fresearch%2Fparticipants%2Fdata%2Fref%2Fh2020%2Fgrants_manual%2Fhi%2Foa_pilot%2Fh2020-hi-oa-data-mgt_en.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NIHDataSharing03-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NIHDataSharing03_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/grants.nih.gov\/grants\/policy\/data_sharing\/data_sharing_guidance.htm\" target=\"_blank\">\"NIH Data Sharing Policy and Implementation Guidance\"<\/a>. <i>Grants and Funding<\/i>. National Institutes of Health. 05 March 2003<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/grants.nih.gov\/grants\/policy\/data_sharing\/data_sharing_guidance.htm\" target=\"_blank\">https:\/\/grants.nih.gov\/grants\/policy\/data_sharing\/data_sharing_guidance.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NIH+Data+Sharing+Policy+and+Implementation+Guidance&rft.atitle=Grants+and+Funding&rft.date=05+March+2003&rft.pub=National+Institutes+of+Health&rft_id=https%3A%2F%2Fgrants.nih.gov%2Fgrants%2Fpolicy%2Fdata_sharing%2Fdata_sharing_guidance.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WTGuidance14-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WTGuidance14_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web-beta.archive.org\/web\/20141018165611\/http:\/\/www.wellcome.ac.uk\/About-us\/Policy\/Spotlight-issues\/Data-sharing\/Guidance-for-researchers\/index.htm\" target=\"_blank\">\"Guidance for researchers: Developing a data management and sharing plan\"<\/a>. <i>Policy and position statements<\/i>. Wellcome Trust. 2014. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.wellcome.ac.uk\/About-us\/Policy\/Spotlight-issues\/Data-sharing\/Guidance-for-researchers\/index.htm\" target=\"_blank\">the original<\/a> on 18 October 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web-beta.archive.org\/web\/20141018165611\/http:\/\/www.wellcome.ac.uk\/About-us\/Policy\/Spotlight-issues\/Data-sharing\/Guidance-for-researchers\/index.htm\" target=\"_blank\">https:\/\/web-beta.archive.org\/web\/20141018165611\/http:\/\/www.wellcome.ac.uk\/About-us\/Policy\/Spotlight-issues\/Data-sharing\/Guidance-for-researchers\/index.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Guidance+for+researchers%3A+Developing+a+data+management+and+sharing+plan&rft.atitle=Policy+and+position+statements&rft.date=2014&rft.pub=Wellcome+Trust&rft_id=https%3A%2F%2Fweb-beta.archive.org%2Fweb%2F20141018165611%2Fhttp%3A%2F%2Fwww.wellcome.ac.uk%2FAbout-us%2FPolicy%2FSpotlight-issues%2FData-sharing%2FGuidance-for-researchers%2Findex.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AGNHMRC_AustralianCode07-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AGNHMRC_AustralianCode07_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nhmrc.gov.au\/guidelines-publications\/r39\" target=\"_blank\">\"Australian Code for the Responsible Conduct of Research\"<\/a>. National Health and Medical Research Council, Australia. 2007<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nhmrc.gov.au\/guidelines-publications\/r39\" target=\"_blank\">https:\/\/www.nhmrc.gov.au\/guidelines-publications\/r39<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Australian+Code+for+the+Responsible+Conduct+of+Research&rft.atitle=&rft.date=2007&rft.pub=National+Health+and+Medical+Research+Council%2C+Australia&rft_id=https%3A%2F%2Fwww.nhmrc.gov.au%2Fguidelines-publications%2Fr39&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NatureDPs-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NatureDPs_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nature.com\/sdata\/policies\/data-policies\" target=\"_blank\">\"Data Policies\"<\/a>. <i>Nature<\/i>. Macmillan Publishers Limited<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nature.com\/sdata\/policies\/data-policies\" target=\"_blank\">https:\/\/www.nature.com\/sdata\/policies\/data-policies<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+Policies&rft.atitle=Nature&rft.pub=Macmillan+Publishers+Limited&rft_id=https%3A%2F%2Fwww.nature.com%2Fsdata%2Fpolicies%2Fdata-policies&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GigaScienceResearch-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GigaScienceResearch_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20140515012736\/http:\/\/www.gigasciencejournal.com:80\/authors\/instructions\/research\" target=\"_blank\">\"Instructions for authors: Research Articles\"<\/a>. <i>GigaScience<\/i>. BioMed Central Ltd. 2014. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.gigasciencejournal.com:80\/authors\/instructions\/research\" target=\"_blank\">the original<\/a> on 15 May 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20140515012736\/http:\/\/www.gigasciencejournal.com:80\/authors\/instructions\/research\" target=\"_blank\">https:\/\/web.archive.org\/web\/20140515012736\/http:\/\/www.gigasciencejournal.com:80\/authors\/instructions\/research<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Instructions+for+authors%3A+Research+Articles&rft.atitle=GigaScience&rft.date=2014&rft.pub=BioMed+Central+Ltd&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20140515012736%2Fhttp%3A%2F%2Fwww.gigasciencejournal.com%3A80%2Fauthors%2Finstructions%2Fresearch&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HaugMetabo13-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HaugMetabo13_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Haug, K.; Salek, R.M.; Conesa, P. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531110\" target=\"_blank\">\"MetaboLights--An open-access general-purpose repository for metabolomics studies and associated meta-data\"<\/a>. <i>Nucleic Acids Research<\/i> <b>41<\/b> (D1): D781-6. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgks1004\" target=\"_blank\">10.1093\/nar\/gks1004<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3531110\/\" target=\"_blank\">PMC3531110<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23109552\" target=\"_blank\">23109552<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531110\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531110<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MetaboLights--An+open-access+general-purpose+repository+for+metabolomics+studies+and+associated+meta-data&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Haug%2C+K.%3B+Salek%2C+R.M.%3B+Conesa%2C+P.&rft.au=Haug%2C+K.%3B+Salek%2C+R.M.%3B+Conesa%2C+P.&rft.date=2013&rft.volume=41&rft.issue=D1&rft.pages=D781-6&rft_id=info:doi\/10.1093%2Fnar%2Fgks1004&rft_id=info:pmc\/PMC3531110&rft_id=info:pmid\/23109552&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3531110&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UCSDMetab-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UCSDMetab_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.metabolomicsworkbench.org\/\" target=\"_blank\">\"Metabolics Workbench\"<\/a>. University of California San Diego<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.metabolomicsworkbench.org\/\" target=\"_blank\">http:\/\/www.metabolomicsworkbench.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Metabolics+Workbench&rft.atitle=&rft.pub=University+of+California+San+Diego&rft_id=http%3A%2F%2Fwww.metabolomicsworkbench.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Python-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Python_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.python.org\/\" target=\"_blank\">\"Python\"<\/a>. Python Software Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.python.org\/\" target=\"_blank\">https:\/\/www.python.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Python&rft.atitle=&rft.pub=Python+Software+Foundation&rft_id=https%3A%2F%2Fwww.python.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Django-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Django_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.djangoproject.com\/\" target=\"_blank\">\"Django\"<\/a>. Django Software Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.djangoproject.com\/\" target=\"_blank\">https:\/\/www.djangoproject.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Django&rft.atitle=&rft.pub=Django+Software+Foundation&rft_id=https%3A%2F%2Fwww.djangoproject.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PostgreSQL-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PostgreSQL_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.postgresql.org\/\" target=\"_blank\">\"PostgreSQL\"<\/a>. PostgreSQL Global Development Group<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.postgresql.org\/\" target=\"_blank\">https:\/\/www.postgresql.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=PostgreSQL&rft.atitle=&rft.pub=PostgreSQL+Global+Development+Group&rft_id=https%3A%2F%2Fwww.postgresql.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MySQL-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MySQL_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.mysql.com\/\" target=\"_blank\">\"MySQL\"<\/a>. Oracle Corporation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.mysql.com\/\" target=\"_blank\">https:\/\/www.mysql.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MySQL&rft.atitle=&rft.pub=Oracle+Corporation&rft_id=https%3A%2F%2Fwww.mysql.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-wxWidgets-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-wxWidgets_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wxwidgets.org\/\" target=\"_blank\">\"wxWidgets\"<\/a>. wxWidgets Development Team<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.wxwidgets.org\/\" target=\"_blank\">https:\/\/www.wxwidgets.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 November 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=wxWidgets&rft.atitle=&rft.pub=wxWidgets+Development+Team&rft_id=https%3A%2F%2Fwww.wxwidgets.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-rsync-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-rsync_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/rsync.samba.org\/\" target=\"_blank\">\"rsync\"<\/a>. Wayne Davison<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/rsync.samba.org\/\" target=\"_blank\">https:\/\/rsync.samba.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=rsync&rft.atitle=&rft.pub=Wayne+Davison&rft_id=https%3A%2F%2Frsync.samba.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScholzSetupX07-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScholzSetupX07_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scholz, M.; Fiehn, O. (2007). \"SetupX -- A public study design database for metabolomic projects\". <i>Pacific Symposium on Biocomputing<\/i> <b>2007<\/b>: 169\u201380. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17990490\" target=\"_blank\">17990490<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SetupX+--+A+public+study+design+database+for+metabolomic+projects&rft.jtitle=Pacific+Symposium+on+Biocomputing&rft.aulast=Scholz%2C+M.%3B+Fiehn%2C+O.&rft.au=Scholz%2C+M.%3B+Fiehn%2C+O.&rft.date=2007&rft.volume=2007&rft.pages=169%E2%80%9380&rft_id=info:pmid\/17990490&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SkogersonTheVol11-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SkogersonTheVol11_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Skogerson, K.; Wohlgemuth, G.; Barupal, D.K.; Fiehn, O. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3199763\" target=\"_blank\">\"The volatile compound BinBase mass spectral database\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b>: 321. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-321\" target=\"_blank\">10.1186\/1471-2105-12-321<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3199763\/\" target=\"_blank\">PMC3199763<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21816034\" target=\"_blank\">21816034<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3199763\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3199763<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+volatile+compound+BinBase+mass+spectral+database&rft.jtitle=BMC+Bioinformatics&rft.aulast=Skogerson%2C+K.%3B+Wohlgemuth%2C+G.%3B+Barupal%2C+D.K.%3B+Fiehn%2C+O.&rft.au=Skogerson%2C+K.%3B+Wohlgemuth%2C+G.%3B+Barupal%2C+D.K.%3B+Fiehn%2C+O.&rft.date=2011&rft.volume=12&rft.pages=321&rft_id=info:doi\/10.1186%2F1471-2105-12-321&rft_id=info:pmc\/PMC3199763&rft_id=info:pmid\/21816034&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3199763&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZolnaiProject03-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZolnaiProject03_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zolnai, Zsolt; Lee, Peter T.; Li, Jing; Chapman, Michael R.; Newman, Craig S.; Phillips Jr., George N.; Rayment, Ivan; Ulrich, Eldon L.; Volkman, Brian F.; Markley, John L. (January 2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.springerlink.com\/content\/p3u654x38832uv73\/fulltext.pdf\" target=\"_blank\">\"Project management system for structural and functional proteomics: Sesame\"<\/a> (PDF). <i>Journal of Structural and Functional Genomics<\/i> <b>4<\/b> (1): 11\u201323. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1023%2FA%3A1024684404761\" target=\"_blank\">10.1023\/A:1024684404761<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.springerlink.com\/content\/p3u654x38832uv73\/fulltext.pdf\" target=\"_blank\">http:\/\/www.springerlink.com\/content\/p3u654x38832uv73\/fulltext.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Project+management+system+for+structural+and+functional+proteomics%3A+Sesame&rft.jtitle=Journal+of+Structural+and+Functional+Genomics&rft.aulast=Zolnai%2C+Zsolt%3B+Lee%2C+Peter+T.%3B+Li%2C+Jing%3B+Chapman%2C+Michael+R.%3B+Newman%2C+Craig+S.%3B+Phillips+Jr.%2C+George+N.%3B+Rayment%2C+Ivan%3B+Ulrich%2C+Eldon+L.%3B+Volkman%2C+Brian+F.%3B+Markley%2C+John+L.&rft.au=Zolnai%2C+Zsolt%3B+Lee%2C+Peter+T.%3B+Li%2C+Jing%3B+Chapman%2C+Michael+R.%3B+Newman%2C+Craig+S.%3B+Phillips+Jr.%2C+George+N.%3B+Rayment%2C+Ivan%3B+Ulrich%2C+Eldon+L.%3B+Volkman%2C+Brian+F.%3B+Markley%2C+John+L.&rft.date=January+2003&rft.volume=4&rft.issue=1&rft.pages=11%E2%80%9323&rft_id=info:doi\/10.1023%2FA%3A1024684404761&rft_id=http%3A%2F%2Fwww.springerlink.com%2Fcontent%2Fp3u654x38832uv73%2Ffulltext.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Markley-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Markley_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Markley, J.L.; Anderson, M.E.; Cui, Q. et al. (2007). \"New bioinformatics resources for metabolomics\". <i>Pacific Symposium on Biocomputing<\/i> <b>2007<\/b>: 157\u2013168. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17990489\" target=\"_blank\">17990489<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=New+bioinformatics+resources+for+metabolomics&rft.jtitle=Pacific+Symposium+on+Biocomputing&rft.aulast=Markley%2C+J.L.%3B+Anderson%2C+M.E.%3B+Cui%2C+Q.+et+al.&rft.au=Markley%2C+J.L.%3B+Anderson%2C+M.E.%3B+Cui%2C+Q.+et+al.&rft.date=2007&rft.volume=2007&rft.pages=157%E2%80%93168&rft_id=info:pmid\/17990489&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoreInfor-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CoreInfor_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.coreinformatics.com\/\" target=\"_blank\">\"Core Informatics\"<\/a>. Core Informatics, LLC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.coreinformatics.com\/\" target=\"_blank\">https:\/\/www.coreinformatics.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved October 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Core+Informatics&rft.atitle=&rft.pub=Core+Informatics%2C+LLC&rft_id=https%3A%2F%2Fwww.coreinformatics.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BioCrates-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BioCrates_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biocrates.com\/\" target=\"_blank\">\"Biocrates Life Sciences\"<\/a>. Biocrates Life Sciences AG<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.biocrates.com\/\" target=\"_blank\">http:\/\/www.biocrates.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved October 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Biocrates+Life+Sciences&rft.atitle=&rft.pub=Biocrates+Life+Sciences+AG&rft_id=http%3A%2F%2Fwww.biocrates.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ClarityLIMS-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ClarityLIMS_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.genologics.com\/editions\/clarity-lims-gold\/\" target=\"_blank\">\"Clarity LIMS Gold\"<\/a>. GenoLogics Life Sciences Software Inc.<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.genologics.com\/editions\/clarity-lims-gold\/\" target=\"_blank\">https:\/\/www.genologics.com\/editions\/clarity-lims-gold\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved October 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Clarity+LIMS+Gold&rft.atitle=&rft.pub=GenoLogics+Life+Sciences+Software+Inc.&rft_id=https%3A%2F%2Fwww.genologics.com%2Feditions%2Fclarity-lims-gold%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article's references were in alphabetical order; the references here are shown in order of appearance in the article due to the way the wiki processes references. In one case the original URL had changed; an archived version of the URL was used instead. In two other cases, the original authors inadvertently used an incorrect reference (for Sesame LIMS and for Java CORBA); a correct reference was added to each for this version. The supplemental figures mentioned in the text have a different name than the ones supplied at the end; a best-effort attempt has been made to match the supplemental figure numbers in the text to the files linked in the \"Supplementary material\" section. (Though S9B and S9C are mentioned, they don't seem to be included, unless they are referencing S9A again.) Some grammar and spelling corrections were also made.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181457\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.512 seconds\nReal time usage: 0.533 seconds\nPreprocessor visited node count: 17393\/1000000\nPreprocessor generated node count: 32311\/1000000\nPost\u2010expand include size: 112217\/2097152 bytes\nTemplate argument size: 39773\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 503.306 1 - -total\n 82.76% 416.522 1 - Template:Reflist\n 71.44% 359.567 25 - Template:Citation\/core\n 56.09% 282.319 20 - Template:Cite_web\n 19.83% 99.783 5 - Template:Cite_journal\n 12.18% 61.289 1 - Template:Infobox_journal_article\n 11.61% 58.414 1 - Template:Infobox\n 6.66% 33.540 80 - Template:Infobox\/row\n 4.55% 22.885 32 - Template:Citation\/make_link\n 2.99% 15.024 9 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10080-0!*!0!!en!5!* and timestamp 20181214181457 and revision id 31042\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics\">https:\/\/www.limswiki.org\/index.php\/Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","c1c21b43f0762acc3ad8147824a0e651_images":["https:\/\/www.limswiki.org\/images\/6\/6a\/Fig1_Hunter_Metabolomics2017_13-2.gif"],"c1c21b43f0762acc3ad8147824a0e651_timestamp":1544811297,"9e160b0ad749c44fbc6303a7527b6e1e_type":"article","9e160b0ad749c44fbc6303a7527b6e1e_title":"Baobab Laboratory Information Management System: Development of an open-source laboratory information management system for biobanking (Bendou et al. 2017)","9e160b0ad749c44fbc6303a7527b6e1e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking","9e160b0ad749c44fbc6303a7527b6e1e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Baobab Laboratory Information Management System: Development of an open-source laboratory information management system for biobanking\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nBaobab Laboratory Information Management System: Development of an open-source laboratory information management system for biobankingJournal\n \nBiopreservation and BiobankingAuthor(s)\n \nBendou, Hocine; Sizani, Lunga; Reid, Tim; Swanepoel, Carmen; Ademuyiwa, Toluwaleke; Merino-Martinez, Roxana; Meuller, Heimo; Abayomi, Akin; Christoffels, AlanAuthor affiliation(s)\n \nUniversity of the Western Cape, B3Africa Consortium, Tygerberg Hospital, Stellenbosch University, H3Africa Consortium, Karolinska Institutet, BBMRI-ERIC Common Service ITPrimary contact\n \nEmail: alan at sanbi dot ac dot zaYear published\n \n2017Volume and issue\n \n15 (2)Page(s)\n \n116-120DOI\n \n10.1089\/bio.2017.0014ISSN\n \n1947-5543Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/online.liebertpub.com\/doi\/full\/10.1089\/bio.2017.0014Download\n \nhttp:\/\/online.liebertpub.com\/doi\/pdfplus\/10.1089\/bio.2017.0014 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Implementation \n4 System architecture \n\n4.1 Hardware and software \n4.2 Workflow description \n4.3 Workflow implementation \n4.4 Kit assembly \n4.5 Shipping \n4.6 Storage management \n4.7 Freezer management \n4.8 Security and administration management \n4.9 Analysis request \n4.10 Availability of software \n\n\n5 Discussion \n6 Acknowledgements \n7 Author disclosure statement \n8 Supplementary material \n9 References \n10 Notes \n\n\n\nAbstract \nA laboratory information management system (LIMS) is central to the informatics infrastructure that underlies biobanking activities. To date, a wide range of commercial and open-source LIMSs are available, and the decision to opt for one LIMS over another is often influenced by the needs of the biobank clients and researchers, as well as available financial resources. The Baobab LIMS was developed by customizing the Bika LIMS software to meet the requirements of biobanking best practices. The need to implement biobank standard operation procedures as well as stimulate the use of standards for biobank data representation motivated the implementation of Baobab LIMS, an open-source LIMS for biobanking. Baobab LIMS comprises modules for biospecimen kit assembly, shipping of biospecimen kits, storage management, analysis requests, reporting, and invoicing. The Baobab LIMS is based on the Plone web-content management framework. All the system requirements for Plone are applicable to Baobab LIMS, including the need for a server with at least 8 GB RAM and 120 GB hard disk space. Baobab LIMS is a client-server-based system, whereby the end user is able to access the system securely through the internet on a standard web browser, thereby eliminating the need for standalone installations on all machines.\n\nIntroduction \nHuman biobanking refers to the collection, processing, and storage of biospecimens and the collection of associated demographic and clinical data for future research use. The extensive collections of biospecimens throughout Africa collected for either specific research, population studies, or part of normal diagnostics workup were not necessarily collected for prospective use by researchers and practitioners. As a result, such collections might not necessarily have followed or adhered to evolving bioethical paradigms and international biobanking best practices.[1][2]\nHowever, the establishment of the concept of centralized biobanks across Africa through initiatives such as H3Africa (http:\/\/www.h3africa.org\/consortium\/projects), the AIDS Cancer Specimen Resource (ACSR; http:\/\/oham.cancer.gov\/oham_research\/programs\/specimen_resource), and the B3Africa (http:\/\/www.b3africa.org) projects has highlighted the need for establishing and harmonizing national and regional biobank governance frameworks to address a relatively unregulated access to human and other ecological samples of academic interest in Africa. At the same time, these governance frameworks fall in line with rapidly changing biobanking practices driven by modern technology.[3][4] Similarly, a governance framework for IT infrastructure requirements that underlies a biobank does not exist.\nAccording to the biological material tracking recommendations within the ISBER best practices, a computer-based inventory system should be in place to allow for the tracking and annotation of each incoming biospecimen into the biobank.[1] An LIMS is thus central to the informatics infrastructure that underlies biobanking activities. To date, a wide range of commercial and open-source LIMSs are available, and the decision to opt for one LIMS over another is often influenced by the needs of the biobank clients and researchers, as well as available financial resources.\nThe National Health Laboratory Services (NHLS)\u2014Stellenbosch University Biobank (NSB), a unit associated with the Division of Haematology at the Faculty of Medicine and Health Sciences, was established in 2012 initially through the ACSR project, and subsequently the NIH H3Africa funding initiative, and required options for an LIMS implementation. The only option at the time was to consider a commercial LIMS because of time constraints to meet the growing need for biobanking services in South Africa. However, access to open-source LIMS software allowed us to consider a longer term implementation that would align with our sustainability plans. Bika LIMS[5] and caTissue (now evolved and known as OpenSpecimen[6]) were identified as long-term options based on input from active software developer and user communities. \nBika LIMS, although not specifically made for human biospecimens, is part of the BIKA software ecosystem that includes BIKA Health for healthcare laboratories and Bika Interlab for interlaboratory proficiency testing. Customization of the Bika LIMS software would provide the benefit of inheriting a range of electronic health record functions that are central to establishing a core facility to support personalized medicine research. The recently funded European project, B3Africa, was established to strengthen IT infrastructure and ethical governance frameworks that would bridge biobanking and biomedical research across Europe and Africa. This funding provided the impetus to revisit the biobanking IT infrastructure at the NSB and to accelerate the development of Baobab LIMS, an open-source LIMS for biobanking, as a strategy to provide a harmonized LIMS as an option for Africa.\nA functional specification exercise[7] in 2013 within the context of NSB biobanking requirements identified the following key modules as part of the extension to the existing Bika LIMS software, namely biospecimen kit assembly, shipping of biospecimen kits, storage management, analysis requests, reporting, and invoicing.\n\nImplementation \nStandard operating procedures (SOPs) associated with biological material inventory management were developed by the NSB team to inform LIMS workflow development. Other SOPs focusing on shipping, labeling, biospecimen procedures, and quality control were also developed in association with other H3Africa biobanks, and they are publicly available (http:\/\/h3africa.org\/consortium\/documents). The collection of SOPs underlies the NSB flowchart of biobank activities (Fig. 1) and subsequently the quality management system. Importantly, even with the use of electronic systems, it is important to keep hard copies of all documentation detailing the biospecimen passage from reception throughout storage to dissemination as a QC check.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Flowchart of the activities within the NSB biobank, showing the activities within the NSB Biobank starting with a face-to-face meeting with a client to the final storage of the biospecimen and\/or the analyses associated with that biospecimen\n\n\n\nSystem architecture \nHardware and software \nBaobab LIMS is based on the Plone web-content management framework (https:\/\/plone.org). All the requirements for Plone are applicable to the Baobab LIMS. As such, the Baobab LIMS is a server\u2013client-based system, whereby the end user is able to access the system through the internet on a standard web browser, thereby eliminating disruptive workstation installations and inefficient software maintenance.\n\nWorkflow description \nThe following processes and tasks were carried out by the NSB and the client to ensure that a biobank project is successfully implemented (Fig. 1):\n(a) The biobank laboratory manager meets with one or more client representatives to decide on the requirements of a new biobank project. A client does not refer to a physical person but rather to an institution such as a laboratory, a hospital, or even another biobank that can be represented by one or more representatives. In the Baobab LIMS system, they are called \"Client Contacts.\" The project requires both parties to define the specimen types that will be collected and the biological analyses that are required at the biobank for all collected specimens. (b) The biobank prepares and assembles the kits that include equipment needed by the client to perform the sample collection, with one kit per participant. Mostly, a kit contains empty barcoded tubes and a form with a field for participant identifier, generated by the client. The collection forms are pre-barcoded with space for the client to use different barcodes for the tubes. In this way, the design of the forms minimizes the risk of assigning the incorrect tube to a participant. (c) The biobank ships the kits to the client by making use of external courier services. (d) The client receives the kits, collects the specimens in the associated tubes, and returns the kits to the biobank. (e) The biobank receives the kits from the client and registers the biospecimens in the system. (f) The biospecimens are then aliquoted for downstream analyses (the analyses defined in the project) or stored in freezers for later analysis.\n\nWorkflow implementation \nThe Boabab LIMS modules were designed to suit most laboratory specimen reception workflows, with minimal customization and postinstallation. In the NSB biobank, these modules were adapted to suit the specific needs as set out in the SOPs that were developed. These SOPs described in detail the life cycle of a specimen as it moves through the laboratory, and which steps require specific documentation, such as the time the samples arrived in the laboratory, the time at which sample accessioning took place, which user was performing the analysis, the method used in the analysis request (linked to its own SOP), and the time of completion of the task, to name a few. In this way, the LIMS was able to adapt to almost all of the manual documentation processes that occur in this specific workflow.\nClient and analysis request modules were imported from Bika LIMS into Baobab LIMS and customized to accommodate human biospecimen laboratory services. The following modules were custom built to support the routine activities of NSB (Fig. 1).\n\nKit assembly \nThe biospecimen \"Kit assembly\" module provides the protocol needed to assemble kits that will be used to collect biospecimens in the field (see Supplementary Data and Supplementary Figs. S1\u2013S5). This module ensures that clients can order the appropriate kits for their biospecimen collection. The project that is registered at NSB will define the appropriate collection kit(s). For example, if a project will collect samples for DNA and RNA extraction at the client's laboratory, then the appropriate collection tubes with an assigned function and label must be shipped to the client for immediate implementation in the field or for storage under appropriate conditions until it is needed. The NSB maintains an inventory of kit components that can be put together and packaged for different sampling procedures and sample types depending on the client's need.\nThe kits include packaging material (styrofoam containers and corrugated shipping carton, absorbent material, and press lock biohazard plastic bag), gloves (optional), documentation (biospecimen submission form containing minimal information data, a shipping manifest [see Supplementary Data and Supplementary Fig. S6], shipping checklist, shipping query form, and a workflow on how to pack and ship the kit), and the specific biospecimen collection tubes that are already labeled. For example, a kit required for DNA and RNA sampling might contain a 6\u2009mL EDTA BD vacutainer or an 8.5\u2009mL PaxGene Blood DNA tube and a 10\u2009mL PaxGene Blood RNA tube or Tempus Blood RNA tubes depending on the downstream application. The kit assembly module provides the user the ability to select the appropriate kit template such as the templates for DNA or RNA sampling, which in turn is used to define the specific SOP required to assemble the material needed for the client's project. The kit template can be specific to one project or reused in different projects.\n\nShipping \nThe shipping module ensures that the correct instructions are given to send the appropriate biospecimen containers (as packaged in the kits) to the client and subsequent e-mail notification to alert the client of incoming kits. Similarly, an e-mail notification based on the LIMS instructions will be sent to notify the NSB upon return of the kits from the client. The NSB, in consultation with the client, defines the appropriate containers to ship to the client and the shipping instructions to and from the client (see Supplementary Data and Supplementary Figs. S6\u2013S10). The kit, assembled in a size-appropriate box (see Supplementary Data and Supplementary Figs. S4 and S5), includes a manifest that describes each kit and their associated barcodes (see Supplementary Data and Supplementary Fig. S6).\n\nStorage management \nBiospecimen, kits, and stock items are handled within the inventory management module. The inventory management module describes the steps needed for the storage location so that products can be ordered and stocks updated accordingly (see Supplementary Data and Supplementary Figs. S10\u2013S16). Biospecimen storage covers a hierarchy of storage levels: storage unit (or room), freezers, shelves, and boxes. The biospecimens are stored in cryoboxes in various sizes depending on the size of the collection tube and cryotubes. For the other storage types (kits and stocks), there is no hierarchy to respect, and the positions can be created at any level.\n\nFreezer management \nDifferent kinds of storage for biospecimens and aliquots exist inside the NSB depending on biospecimen type as well as the need for short- or long-term storage. The freezer management module describes the steps needed to define the structure that matches the physical storage in the NSB: rooms (within rooms), freezers, shelves, cryoboxes, and positions\/locations (see Supplementary Data and Supplementary Figs. S17\u2013S20). A freezer contains shelves that contain cryoboxes. The last storage level at the NSB, namely cryoboxes, contains the positions reserved for biospecimens and aliquots. Similarly, the liquid nitrogen freezer and\/or dewar contain, respectively, racks that contain canes. The box and\/or cane can have multiple positions for biospecimen storage. Three classes (content types) were used to design the freezer management module (see Supplementary Data and Supplementary Figs. S18 and S21), namely storage unit (room), storage level (freezer, shelf, and cryobox), and storage location (positions inside cryobox). This class inheritance was implemented using the object database ZODB (http:\/\/www.zodb.org\/en\/latest).\n\nSecurity and administration management \nPlone (the web-content management platform for Bika software) is based on the Zope framework. Zope provides built-in security functionality that allows us to define roles with permissions. A \"permission\" controls whether logged-in or anonymous users with a specific role can execute code and access contents. Each NSB staff member and client will have specific assigned roles and a level of security. Baobab LIMS also inherits Plone's secure version-controlled document management system and reviewer workflows and audit trails.\nPlone uses ZODB to store user data. ZODB, compared with SQL-based databases, is not vulnerable to injection as it uses binary format that cannot have user data inserted. Plone authenticates users in its own database using a salted secure hash algorithm (SSHA) hash of their password. Using its modular authentication system, Plone can also authenticate users against common authentication systems such as LDAP and SQL as well as any other system for which a plugin is available (Gmail, OpenID, etc.). After authentication, Plone creates a session using an SHA-256 hash of a secret stored on the server, the userid, and the current time. This is based on the Apache auth_tkt cookie format, but with a more secure hash function.\n\nAnalysis request \nA global list of available analyses is defined by the capabilities of the biobank laboratory. At the project level, analyses are defined for biospecimens based on the requirements of the project, such as DNA and\/or RNA extraction applied to blood samples, and the resulting quality and purity results of the extracted DNA and\/or RNA. Results of these analyses are registered and reported to the client (see Supplementary Figs. S22 and S23). The data are imported into Baobab LIMS through an instrument interface such as the biodrop interface (see Supplementary Data and Supplementary Fig. S24).\n\nAvailability of software \nSource code is available directly from (https:\/\/github.com\/hocinebendou\/baobab.lims.git) or (http:\/\/christoffels.sanbi.ac.za\/index.php\/software\/software-downloads).\nA demo site has been configured at b3abiobank.sanbi.ac.za\/demo (login\u2009=\u2009admin; password\u2009=\u2009admin).\n\nDiscussion \nHigh-throughput genetic tools allow researchers to rapidly analyze thousands of biospecimens, as is common in consortia focused on population-based studies. In response, biobanks have to carefully consider the appropriate IT infrastructure that can meet the demands of large genetics studies. The Baobab LIMS was designed as an open-source alternative for use in a resource-limited setting to meet the demands of increasing biospecimen collection.\nDuring the development of Baobab LIMS, a new open-source LIMS, Acquire, was published with a focus on pathology biospecimens.[8] This tool integrates the inventory management functionality of OpenSpecimen with modules specifically designed to meet requirements of researchers in a pathology laboratory. However, tools such as OpenSpecimen do not handle the specific biobank activities of NSB, which includes supporting prepackaging of biospecimen laboratory kits for clients (Fig. 1).\nThe establishment of centralized biobanks in Africa has drawn attention to the issues of interoperability between biobanks and harmonization of terminology used by each biobank. These concerns are not unique to African biobanks.[9][10] Data standards such as MIABIS[11] provide an ideal platform to integrate data among biobanks. The advantage of using the same terminology across biobanks was demonstrated through the development of the virtual Breast Cancer Campaign Tissue Bank.[12][13] Although a biobank catalogue is not currently available at NSB, we envisage a simplified application programming interface (API) that would allow users to access summary information using data stored in Baobab LIMS.\nQinlan et al.[13] suggested an increased role of ethical boards, governance, accreditation bodies, and funders to ensure that groups being authorized to collect samples have sufficient informatics capabilities to ensure the samples are used. This suggestion ensures that teams authorized to collect samples will also have the technical skills to ensure that the associated data are managed correctly. This concept relates to issues of biobank sustainability. Technical costs at a biobank go beyond supporting an LIMS and instead should incorporate a biobanking informatics management system that allows the biobank the flexibility to meet the growing technical demands of making data available to a wider scientific community beyond biospecimen handling in a laboratory. In this context, and doubling the cost suggested by Dowst et al.[8] for maintaining their Acquire LIMS, the cost of employing a linux administrator (20% full-time equivalent [FTE]), database administrator (10% FTE) and a computer programmer (25% FTE) in a South African setting would cost $30,000 U.S. The demands placed on IT staff require that biobanks have dedicated staff for these technical roles, thereby increasing personnel costs at least fivefold. These costs are a key consideration for centralizing and harmonizing biobanking in Africa. Unfortunately, local, regional, and international funders need to appreciate the importance of this critical component required for modern biobanking and the academic advantage of interoperability across evolving biobanks.\nThe adoption of Baobab LIMS by a wider user community will require more generic and configurable workflows. Nevertheless, Baobab LIMS has become a central component in the eB3Kit of the B3Africa H2020 project and is also included in the BBMRI-ERIC software catalogue, and in the open BIBBOX application store.[14]\n\nAcknowledgements \nThis project has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement 654404 and was supported by the South African Research Chairs Initiative of the Department of Science and Technology and National Research Foundation of South Africa. This work was also supported by the H3Africa UH2\/UH3 Biorepository grant (IUH2HG007092-01 to A.A) from the National Institutes of Health (NIH). We also thank the Department of Science and Technology (DST), National Health Laboratory Services (NHLS), and Stellenbosch University (SU) for their support. Thanks to Lemoene Smit and Campbell McKellar-Bassett for their technical input and consultation to customize Bika LIMS.\n\nAuthor disclosure statement \nNo conflicting financial interests exist.\n\nSupplementary material \nSupplementary Data and Figures\n\nReferences \n\n\n\u2191 1.0 1.1 International Society for Biological and Environmental Repositories (2012). \"2012 Best Practices for Repositories: Collection, Storage, Retrieval, and Distribution of Biological Materials for Research\". Biopreservation and Biobanking 10 (2): 79-161. doi:10.1089\/bio.2012.1022. PMID 24844904.   \n\n\u2191 Abayomi, A.; Christoffels, A.; Grewal, R. et al. (2013). \"Challenges of biobanking in South Africa to facilitate indigenous research in an environment burdened with human immunodeficiency virus, tuberculosis, and emerging noncommunicable diseases\". Biopreservation and Biobanking 11 (6): 347\u2013354. doi:10.1089\/bio.2013.0049. PMID 24835364.   \n\n\u2191 Dhai, A. (2013). \"Establishing national biobanks in South Africa: The urgent need for an ethico-regulatory framework\". The South African Journal of Bioethics & Law 6 (2): 38\u201339. doi:10.7196\/SAJBL.296.   \n\n\u2191 de Vries, J.; Abayomi, A.; Brandful, J. et al. (2014). \"A perpetual source of DNA or something really different: ethical issues in the creation of cell lines for African genomics research\". BMC Medical Ethics 15: 60. doi:10.1186\/1472-6939-15-60.   \n\n\u2191 \"BIKA\". Bika Lab Systems (Pty) Ltd. https:\/\/www.bikalims.org\/ . Retrieved 16 August 2016 .   \n\n\u2191 \"OpenSpecimen\". Krishagni Solutions Pvt. Ltd. http:\/\/www.openspecimen.org\/ . Retrieved 16 August 2016 .   \n\n\u2191 NCB-H3A (23 March 2013). \"NCB-H3A Cape Town Biobank: Biobank Information Management System - Functional Requirements Overview & Phase I Objectives\". South African National Bioinformatics Institute. http:\/\/christoffels.sanbi.ac.za\/index.php\/projects\/biobanking . Retrieved 16 August 2016 .   \n\n\u2191 8.0 8.1 Dowst, H.; Pew, B.; Watkins, C. et al. (2015). \"Acquire: An open-source comprehensive cancer biobanking system\". Bioinformatics 31 (10): 1655\u201362. doi:10.1093\/bioinformatics\/btv012. PMC PMC4426840. PMID 25573920. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4426840 .   \n\n\u2191 Fearn, P.; Michels, C.; Meagher, K.; Cada, M. (2013). \"2012 International Society for Biological and Environmental Repositories Informatics Working Group: Survey results and conclusions\". Biopreservation and Biobanking 11 (1): 64-6. doi:10.1089\/bio.2012.1115. PMID 24845257.   \n\n\u2191 Quality Information Committee (April 2013). \"First National Data Quality Review: Executive Summary\" (PDF). NHS England. https:\/\/www.england.nhs.uk\/wp-content\/uploads\/2013\/04\/1ndqr-exec-sum.pdf . Retrieved 19 August 2016 .   \n\n\u2191 Merino-Martinez, R.; Norlin, L.; van Enckevort, D. et al. (2016). \"Toward Global Biobank Integration by Implementation of the Minimum Information About BIobank Data Sharing (MIABIS 2.0 Core)\". Biopreservation and Biobanking 14 (4): 298-306. doi:10.1089\/bio.2015.0070. PMID 26977825.   \n\n\u2191 \"Breast Cancer Campaign Tissue Bank - Search\". Breast Cancer Campaign. Archived from the original on 01 August 2014. https:\/\/web.archive.org\/web\/20140801214940\/https:\/\/breastcancertissuebank.org\/bcc\/tissueBank?Name=sampleFinder . Retrieved 19 August 2016 .   \n\n\u2191 13.0 13.1 Quinlan, P.R.; Groves, M.; Jordan, L.B. et al. (2015). \"The Informatics Challenges Facing Biobanks: A Perspective from a United Kingdom Biobanking Network\". Biopreservation and Biobanking 13 (5): 363-70. doi:10.1089\/bio.2014.0099. PMC PMC4675179. PMID 26418270. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4675179 .   \n\n\u2191 M\u00fcller, H.; Malservet, N.; Quinlan, P. et al. (2017). \"From the evaluation of existing solutions to an all-inclusive package for biobanks\". Health and Technology 7 (1): 89-95. doi:10.1007\/s12553-016-0175-x. PMC PMC5346419. PMID 28344915. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5346419 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The link to the Breast Cancer Campaign Tissue Bank search (reference 12) is broken; an archived version of the link is used instead.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\">https:\/\/www.limswiki.org\/index.php\/Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 10 May 2017, at 17:53.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 813 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","9e160b0ad749c44fbc6303a7527b6e1e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Baobab_Laboratory_Information_Management_System_Development_of_an_open-source_laboratory_information_management_system_for_biobanking skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Baobab Laboratory Information Management System: Development of an open-source laboratory information management system for biobanking<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>A <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management system<\/a> (LIMS) is central to the <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" target=\"_blank\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">informatics<\/a> infrastructure that underlies <a href=\"https:\/\/www.limswiki.org\/index.php\/Biobanking\" title=\"Biobanking\" class=\"mw-redirect wiki-link\" target=\"_blank\" data-key=\"9b28907f4da2412ae15cd25822a250e8\">biobanking<\/a> activities. To date, a wide range of commercial and open-source LIMSs are available, and the decision to opt for one LIMS over another is often influenced by the needs of the biobank clients and researchers, as well as available financial resources. The Baobab LIMS was developed by customizing the <a href=\"https:\/\/www.limswiki.org\/index.php\/Bika_LIMS\" title=\"Bika LIMS\" target=\"_blank\" class=\"wiki-link\" data-key=\"daf3e2e28f594ad55c814ae802d43fcb\">Bika LIMS<\/a> software to meet the requirements of biobanking best practices. The need to implement biobank standard operation procedures as well as stimulate the use of standards for biobank data representation motivated the implementation of Baobab LIMS, an open-source LIMS for biobanking. Baobab LIMS comprises modules for biospecimen kit assembly, shipping of biospecimen kits, storage management, analysis requests, reporting, and invoicing. The Baobab LIMS is based on the Plone web-content management framework. All the system requirements for Plone are applicable to Baobab LIMS, including the need for a server with at least 8 GB RAM and 120 GB hard disk space. Baobab LIMS is a client-server-based system, whereby the end user is able to access the system securely through the internet on a standard web browser, thereby eliminating the need for standalone installations on all machines.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Human biobanking refers to the collection, processing, and storage of biospecimens and the collection of associated demographic and clinical data for future research use. The extensive collections of biospecimens throughout Africa collected for either specific research, population studies, or part of normal diagnostics workup were not necessarily collected for prospective use by researchers and practitioners. As a result, such collections might not necessarily have followed or adhered to evolving bioethical paradigms and international biobanking best practices.<sup id=\"rdp-ebb-cite_ref-ISBER2012_12_1-0\" class=\"reference\"><a href=\"#cite_note-ISBER2012_12-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AbayomiChallenges13_2-0\" class=\"reference\"><a href=\"#cite_note-AbayomiChallenges13-2\" rel=\"external_link\">[2]<\/a><\/sup>\n<\/p><p>However, the establishment of the concept of centralized biobanks across Africa through initiatives such as H3Africa (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.h3africa.org\/consortium\/projects\" target=\"_blank\">http:\/\/www.h3africa.org\/consortium\/projects<\/a>), the AIDS Cancer Specimen Resource (ACSR; <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/oham.cancer.gov\/oham_research\/programs\/specimen_resource\" target=\"_blank\">http:\/\/oham.cancer.gov\/oham_research\/programs\/specimen_resource<\/a>), and the B3Africa (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.b3africa.org\" target=\"_blank\">http:\/\/www.b3africa.org<\/a>) projects has highlighted the need for establishing and harmonizing national and regional biobank governance frameworks to address a relatively unregulated access to human and other ecological samples of academic interest in Africa. At the same time, these governance frameworks fall in line with rapidly changing biobanking practices driven by modern technology.<sup id=\"rdp-ebb-cite_ref-DhaiEstab13_3-0\" class=\"reference\"><a href=\"#cite_note-DhaiEstab13-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-deVriesAPerpetual14_4-0\" class=\"reference\"><a href=\"#cite_note-deVriesAPerpetual14-4\" rel=\"external_link\">[4]<\/a><\/sup> Similarly, a governance framework for IT infrastructure requirements that underlies a biobank does not exist.\n<\/p><p>According to the biological material tracking recommendations within the ISBER best practices, a computer-based inventory system should be in place to allow for the tracking and annotation of each incoming biospecimen into the biobank.<sup id=\"rdp-ebb-cite_ref-ISBER2012_12_1-1\" class=\"reference\"><a href=\"#cite_note-ISBER2012_12-1\" rel=\"external_link\">[1]<\/a><\/sup> An LIMS is thus central to the informatics infrastructure that underlies biobanking activities. To date, a wide range of commercial and open-source LIMSs are available, and the decision to opt for one LIMS over another is often influenced by the needs of the biobank clients and researchers, as well as available financial resources.\n<\/p><p>The National Health Laboratory Services (NHLS)\u2014Stellenbosch University Biobank (NSB), a unit associated with the Division of Haematology at the Faculty of Medicine and Health Sciences, was established in 2012 initially through the ACSR project, and subsequently the NIH H3Africa funding initiative, and required options for an LIMS implementation. The only option at the time was to consider a commercial LIMS because of time constraints to meet the growing need for biobanking services in South Africa. However, access to open-source LIMS software allowed us to consider a longer term implementation that would align with our sustainability plans. Bika LIMS<sup id=\"rdp-ebb-cite_ref-BIKA_5-0\" class=\"reference\"><a href=\"#cite_note-BIKA-5\" rel=\"external_link\">[5]<\/a><\/sup> and <a href=\"https:\/\/www.limswiki.org\/index.php\/National_Cancer_Institute#Technology_and_products\" title=\"National Cancer Institute\" target=\"_blank\" class=\"wiki-link\" data-key=\"adcf613b1434adfecdaf79bc475c3370\">caTissue<\/a> (now evolved and known as <a href=\"https:\/\/www.limswiki.org\/index.php\/OpenSpecimen\" title=\"OpenSpecimen\" target=\"_blank\" class=\"wiki-link\" data-key=\"ede30d8d1185cde6f70bb12423d3d496\">OpenSpecimen<\/a><sup id=\"rdp-ebb-cite_ref-OpenSpecimen_6-0\" class=\"reference\"><a href=\"#cite_note-OpenSpecimen-6\" rel=\"external_link\">[6]<\/a><\/sup>) were identified as long-term options based on input from active software developer and user communities. \n<\/p><p>Bika LIMS, although not specifically made for human biospecimens, is part of the BIKA software ecosystem that includes BIKA Health for healthcare <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratories<\/a> and Bika Interlab for interlaboratory proficiency testing. Customization of the Bika LIMS software would provide the benefit of inheriting a range of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health record<\/a> functions that are central to establishing a core facility to support personalized medicine research. The recently funded European project, B3Africa, was established to strengthen IT infrastructure and ethical governance frameworks that would bridge biobanking and biomedical research across Europe and Africa. This funding provided the impetus to revisit the biobanking IT infrastructure at the NSB and to accelerate the development of Baobab LIMS, an open-source LIMS for biobanking, as a strategy to provide a harmonized LIMS as an option for Africa.\n<\/p><p>A functional specification exercise<sup id=\"rdp-ebb-cite_ref-NCB-H3ACape13_7-0\" class=\"reference\"><a href=\"#cite_note-NCB-H3ACape13-7\" rel=\"external_link\">[7]<\/a><\/sup> in 2013 within the context of NSB biobanking requirements identified the following key modules as part of the extension to the existing Bika LIMS software, namely biospecimen kit assembly, shipping of biospecimen kits, storage management, analysis requests, reporting, and invoicing.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<p>Standard operating procedures (SOPs) associated with biological material inventory management were developed by the NSB team to inform LIMS workflow development. Other SOPs focusing on shipping, labeling, biospecimen procedures, and quality control were also developed in association with other H3Africa biobanks, and they are publicly available (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/h3africa.org\/consortium\/documents\" target=\"_blank\">http:\/\/h3africa.org\/consortium\/documents<\/a>). The collection of SOPs underlies the NSB flowchart of biobank activities (Fig. 1) and subsequently the quality management system. Importantly, even with the use of electronic systems, it is important to keep hard copies of all documentation detailing the biospecimen passage from reception throughout storage to dissemination as a QC check.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Bendou_BiopresAndBiobank2017_15-2.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"2f6d9f9836c600227f0be5f44932d3c8\"><img alt=\"Fig1 Bendou BiopresAndBiobank2017 15-2.gif\" src=\"https:\/\/www.limswiki.org\/images\/7\/7a\/Fig1_Bendou_BiopresAndBiobank2017_15-2.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Flowchart of the activities within the NSB biobank, showing the activities within the NSB Biobank starting with a face-to-face meeting with a client to the final storage of the biospecimen and\/or the analyses associated with that biospecimen<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"System_architecture\">System architecture<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Hardware_and_software\">Hardware and software<\/span><\/h3>\n<p>Baobab LIMS is based on the Plone web-content management framework (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/plone.org\" target=\"_blank\">https:\/\/plone.org<\/a>). All the requirements for Plone are applicable to the Baobab LIMS. As such, the Baobab LIMS is a server\u2013client-based system, whereby the end user is able to access the system through the internet on a standard web browser, thereby eliminating disruptive workstation installations and inefficient software maintenance.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Workflow_description\">Workflow description<\/span><\/h3>\n<p>The following processes and tasks were carried out by the NSB and the client to ensure that a biobank project is successfully implemented (Fig. 1):\n<\/p><p>(a) The biobank laboratory manager meets with one or more client representatives to decide on the requirements of a new biobank project. A client does not refer to a physical person but rather to an institution such as a laboratory, a hospital, or even another biobank that can be represented by one or more representatives. In the Baobab LIMS system, they are called \"Client Contacts.\" The project requires both parties to define the specimen types that will be collected and the biological analyses that are required at the biobank for all collected specimens. (b) The biobank prepares and assembles the kits that include equipment needed by the client to perform the sample collection, with one kit per participant. Mostly, a kit contains empty barcoded tubes and a form with a field for participant identifier, generated by the client. The collection forms are pre-barcoded with space for the client to use different barcodes for the tubes. In this way, the design of the forms minimizes the risk of assigning the incorrect tube to a participant. (c) The biobank ships the kits to the client by making use of external courier services. (d) The client receives the kits, collects the specimens in the associated tubes, and returns the kits to the biobank. (e) The biobank receives the kits from the client and registers the biospecimens in the system. (f) The biospecimens are then aliquoted for downstream analyses (the analyses defined in the project) or stored in freezers for later analysis.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Workflow_implementation\">Workflow implementation<\/span><\/h3>\n<p>The Boabab LIMS modules were designed to suit most laboratory specimen reception workflows, with minimal customization and postinstallation. In the NSB biobank, these modules were adapted to suit the specific needs as set out in the SOPs that were developed. These SOPs described in detail the life cycle of a specimen as it moves through the laboratory, and which steps require specific documentation, such as the time the samples arrived in the laboratory, the time at which sample accessioning took place, which user was performing the analysis, the method used in the analysis request (linked to its own SOP), and the time of completion of the task, to name a few. In this way, the LIMS was able to adapt to almost all of the manual documentation processes that occur in this specific workflow.\n<\/p><p>Client and analysis request modules were imported from Bika LIMS into Baobab LIMS and customized to accommodate human biospecimen laboratory services. The following modules were custom built to support the routine activities of NSB (Fig. 1).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Kit_assembly\">Kit assembly<\/span><\/h3>\n<p>The biospecimen \"Kit assembly\" module provides the protocol needed to assemble kits that will be used to collect biospecimens in the field (see Supplementary Data and Supplementary Figs. S1\u2013S5). This module ensures that clients can order the appropriate kits for their biospecimen collection. The project that is registered at NSB will define the appropriate collection kit(s). For example, if a project will collect samples for DNA and RNA extraction at the client's laboratory, then the appropriate collection tubes with an assigned function and label must be shipped to the client for immediate implementation in the field or for storage under appropriate conditions until it is needed. The NSB maintains an inventory of kit components that can be put together and packaged for different sampling procedures and sample types depending on the client's need.\n<\/p><p>The kits include packaging material (styrofoam containers and corrugated shipping carton, absorbent material, and press lock biohazard plastic bag), gloves (optional), documentation (biospecimen submission form containing minimal information data, a shipping manifest [see Supplementary Data and Supplementary Fig. S6], shipping checklist, shipping query form, and a workflow on how to pack and ship the kit), and the specific biospecimen collection tubes that are already labeled. For example, a kit required for DNA and RNA sampling might contain a 6\u2009mL EDTA BD vacutainer or an 8.5\u2009mL PaxGene Blood DNA tube and a 10\u2009mL PaxGene Blood RNA tube or Tempus Blood RNA tubes depending on the downstream application. The kit assembly module provides the user the ability to select the appropriate kit template such as the templates for DNA or RNA sampling, which in turn is used to define the specific SOP required to assemble the material needed for the client's project. The kit template can be specific to one project or reused in different projects.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Shipping\">Shipping<\/span><\/h3>\n<p>The shipping module ensures that the correct instructions are given to send the appropriate biospecimen containers (as packaged in the kits) to the client and subsequent e-mail notification to alert the client of incoming kits. Similarly, an e-mail notification based on the LIMS instructions will be sent to notify the NSB upon return of the kits from the client. The NSB, in consultation with the client, defines the appropriate containers to ship to the client and the shipping instructions to and from the client (see Supplementary Data and Supplementary Figs. S6\u2013S10). The kit, assembled in a size-appropriate box (see Supplementary Data and Supplementary Figs. S4 and S5), includes a manifest that describes each kit and their associated barcodes (see Supplementary Data and Supplementary Fig. S6).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Storage_management\">Storage management<\/span><\/h3>\n<p>Biospecimen, kits, and stock items are handled within the inventory management module. The inventory management module describes the steps needed for the storage location so that products can be ordered and stocks updated accordingly (see Supplementary Data and Supplementary Figs. S10\u2013S16). Biospecimen storage covers a hierarchy of storage levels: storage unit (or room), freezers, shelves, and boxes. The biospecimens are stored in cryoboxes in various sizes depending on the size of the collection tube and cryotubes. For the other storage types (kits and stocks), there is no hierarchy to respect, and the positions can be created at any level.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Freezer_management\">Freezer management<\/span><\/h3>\n<p>Different kinds of storage for biospecimens and aliquots exist inside the NSB depending on biospecimen type as well as the need for short- or long-term storage. The freezer management module describes the steps needed to define the structure that matches the physical storage in the NSB: rooms (within rooms), freezers, shelves, cryoboxes, and positions\/locations (see Supplementary Data and Supplementary Figs. S17\u2013S20). A freezer contains shelves that contain cryoboxes. The last storage level at the NSB, namely cryoboxes, contains the positions reserved for biospecimens and aliquots. Similarly, the liquid nitrogen freezer and\/or dewar contain, respectively, racks that contain canes. The box and\/or cane can have multiple positions for biospecimen storage. Three classes (content types) were used to design the freezer management module (see Supplementary Data and Supplementary Figs. S18 and S21), namely storage unit (room), storage level (freezer, shelf, and cryobox), and storage location (positions inside cryobox). This class inheritance was implemented using the object database ZODB (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.zodb.org\/en\/latest\" target=\"_blank\">http:\/\/www.zodb.org\/en\/latest<\/a>).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Security_and_administration_management\">Security and administration management<\/span><\/h3>\n<p>Plone (the web-content management platform for Bika software) is based on the Zope framework. Zope provides built-in security functionality that allows us to define roles with permissions. A \"permission\" controls whether logged-in or anonymous users with a specific role can execute code and access contents. Each NSB staff member and client will have specific assigned roles and a level of security. Baobab LIMS also inherits Plone's secure version-controlled document management system and reviewer workflows and audit trails.\n<\/p><p>Plone uses ZODB to store user data. ZODB, compared with SQL-based databases, is not vulnerable to injection as it uses binary format that cannot have user data inserted. Plone authenticates users in its own database using a salted secure hash algorithm (SSHA) hash of their password. Using its modular authentication system, Plone can also authenticate users against common authentication systems such as LDAP and SQL as well as any other system for which a plugin is available (Gmail, OpenID, etc.). After authentication, Plone creates a session using an SHA-256 hash of a secret stored on the server, the <i>userid<\/i>, and the current time. This is based on the Apache auth_tkt cookie format, but with a more secure hash function.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Analysis_request\">Analysis request<\/span><\/h3>\n<p>A global list of available analyses is defined by the capabilities of the biobank laboratory. At the project level, analyses are defined for biospecimens based on the requirements of the project, such as DNA and\/or RNA extraction applied to blood samples, and the resulting quality and purity results of the extracted DNA and\/or RNA. Results of these analyses are registered and reported to the client (see Supplementary Figs. S22 and S23). The data are imported into Baobab LIMS through an instrument interface such as the biodrop interface (see Supplementary Data and Supplementary Fig. S24).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Availability_of_software\">Availability of software<\/span><\/h3>\n<p>Source code is available directly from (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/hocinebendou\/baobab.lims.git\" target=\"_blank\">https:\/\/github.com\/hocinebendou\/baobab.lims.git<\/a>) or (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/christoffels.sanbi.ac.za\/index.php\/software\/software-downloads\" target=\"_blank\">http:\/\/christoffels.sanbi.ac.za\/index.php\/software\/software-downloads<\/a>).\n<\/p><p>A demo site has been configured at b3abiobank.sanbi.ac.za\/demo (login\u2009=\u2009admin; password\u2009=\u2009admin).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>High-throughput genetic tools allow researchers to rapidly analyze thousands of biospecimens, as is common in consortia focused on population-based studies. In response, biobanks have to carefully consider the appropriate IT infrastructure that can meet the demands of large genetics studies. The Baobab LIMS was designed as an open-source alternative for use in a resource-limited setting to meet the demands of increasing biospecimen collection.\n<\/p><p>During the development of Baobab LIMS, a new open-source LIMS, Acquire, was published with a focus on pathology biospecimens.<sup id=\"rdp-ebb-cite_ref-DowstAcquire15_8-0\" class=\"reference\"><a href=\"#cite_note-DowstAcquire15-8\" rel=\"external_link\">[8]<\/a><\/sup> This tool integrates the inventory management functionality of OpenSpecimen with modules specifically designed to meet requirements of researchers in a pathology laboratory. However, tools such as OpenSpecimen do not handle the specific biobank activities of NSB, which includes supporting prepackaging of biospecimen laboratory kits for clients (Fig. 1).\n<\/p><p>The establishment of centralized biobanks in Africa has drawn attention to the issues of interoperability between biobanks and harmonization of terminology used by each biobank. These concerns are not unique to African biobanks.<sup id=\"rdp-ebb-cite_ref-Fearn2012_13_9-0\" class=\"reference\"><a href=\"#cite_note-Fearn2012_13-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NHSEnglandQICExec13_10-0\" class=\"reference\"><a href=\"#cite_note-NHSEnglandQICExec13-10\" rel=\"external_link\">[10]<\/a><\/sup> Data standards such as MIABIS<sup id=\"rdp-ebb-cite_ref-Merino-MartinezrToward16_11-0\" class=\"reference\"><a href=\"#cite_note-Merino-MartinezrToward16-11\" rel=\"external_link\">[11]<\/a><\/sup> provide an ideal platform to integrate data among biobanks. The advantage of using the same terminology across biobanks was demonstrated through the development of the virtual Breast Cancer Campaign Tissue Bank.<sup id=\"rdp-ebb-cite_ref-BCCTBSearchArch_12-0\" class=\"reference\"><a href=\"#cite_note-BCCTBSearchArch-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-QuinlanTheInfo15_13-0\" class=\"reference\"><a href=\"#cite_note-QuinlanTheInfo15-13\" rel=\"external_link\">[13]<\/a><\/sup> Although a biobank catalogue is not currently available at NSB, we envisage a simplified application programming interface (API) that would allow users to access summary information using data stored in Baobab LIMS.\n<\/p><p>Qinlan <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-QuinlanTheInfo15_13-1\" class=\"reference\"><a href=\"#cite_note-QuinlanTheInfo15-13\" rel=\"external_link\">[13]<\/a><\/sup> suggested an increased role of ethical boards, governance, accreditation bodies, and funders to ensure that groups being authorized to collect samples have sufficient informatics capabilities to ensure the samples are used. This suggestion ensures that teams authorized to collect samples will also have the technical skills to ensure that the associated data are managed correctly. This concept relates to issues of biobank sustainability. Technical costs at a biobank go beyond supporting an LIMS and instead should incorporate a biobanking informatics management system that allows the biobank the flexibility to meet the growing technical demands of making data available to a wider scientific community beyond biospecimen handling in a laboratory. In this context, and doubling the cost suggested by Dowst <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-DowstAcquire15_8-1\" class=\"reference\"><a href=\"#cite_note-DowstAcquire15-8\" rel=\"external_link\">[8]<\/a><\/sup> for maintaining their Acquire LIMS, the cost of employing a linux administrator (20% full-time equivalent [FTE]), database administrator (10% FTE) and a computer programmer (25% FTE) in a South African setting would cost $30,000 U.S. The demands placed on IT staff require that biobanks have dedicated staff for these technical roles, thereby increasing personnel costs at least fivefold. These costs are a key consideration for centralizing and harmonizing biobanking in Africa. Unfortunately, local, regional, and international funders need to appreciate the importance of this critical component required for modern biobanking and the academic advantage of interoperability across evolving biobanks.\n<\/p><p>The adoption of Baobab LIMS by a wider user community will require more generic and configurable workflows. Nevertheless, Baobab LIMS has become a central component in the eB3Kit of the B3Africa H2020 project and is also included in the BBMRI-ERIC software catalogue, and in the open BIBBOX application store.<sup id=\"rdp-ebb-cite_ref-M.C3.BCllerFrom17_14-0\" class=\"reference\"><a href=\"#cite_note-M.C3.BCllerFrom17-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This project has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement 654404 and was supported by the South African Research Chairs Initiative of the Department of Science and Technology and National Research Foundation of South Africa. This work was also supported by the H3Africa UH2\/UH3 Biorepository grant (IUH2HG007092-01 to A.A) from the National Institutes of Health (NIH). We also thank the Department of Science and Technology (DST), National Health Laboratory Services (NHLS), and Stellenbosch University (SU) for their support. Thanks to Lemoene Smit and Campbell McKellar-Bassett for their technical input and consultation to customize Bika LIMS.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Author_disclosure_statement\">Author disclosure statement<\/span><\/h2>\n<p>No conflicting financial interests exist.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_material\">Supplementary material<\/span><\/h2>\n<p><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/online.liebertpub.com\/doi\/suppl\/10.1089\/bio.2017.0014\/suppl_file\/Supp_Data.pdf\" target=\"_blank\">Supplementary Data and Figures<\/a>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-ISBER2012_12-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISBER2012_12_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISBER2012_12_1-1\" rel=\"external_link\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">International Society for Biological and Environmental Repositories (2012). \"2012 Best Practices for Repositories: Collection, Storage, Retrieval, and Distribution of Biological Materials for Research\". <i>Biopreservation and Biobanking<\/i> <b>10<\/b> (2): 79-161. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fbio.2012.1022\" target=\"_blank\">10.1089\/bio.2012.1022<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24844904\" target=\"_blank\">24844904<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=2012+Best+Practices+for+Repositories%3A+Collection%2C+Storage%2C+Retrieval%2C+and+Distribution+of+Biological+Materials+for+Research&rft.jtitle=Biopreservation+and+Biobanking&rft.aulast=International+Society+for+Biological+and+Environmental+Repositories&rft.au=International+Society+for+Biological+and+Environmental+Repositories&rft.date=2012&rft.volume=10&rft.issue=2&rft.pages=79-161&rft_id=info:doi\/10.1089%2Fbio.2012.1022&rft_id=info:pmid\/24844904&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbayomiChallenges13-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbayomiChallenges13_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abayomi, A.; Christoffels, A.; Grewal, R. et al. (2013). \"Challenges of biobanking in South Africa to facilitate indigenous research in an environment burdened with human immunodeficiency virus, tuberculosis, and emerging noncommunicable diseases\". <i>Biopreservation and Biobanking<\/i> <b>11<\/b> (6): 347\u2013354. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fbio.2013.0049\" target=\"_blank\">10.1089\/bio.2013.0049<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24835364\" target=\"_blank\">24835364<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Challenges+of+biobanking+in+South+Africa+to+facilitate+indigenous+research+in+an+environment+burdened+with+human+immunodeficiency+virus%2C+tuberculosis%2C+and+emerging+noncommunicable+diseases&rft.jtitle=Biopreservation+and+Biobanking&rft.aulast=Abayomi%2C+A.%3B+Christoffels%2C+A.%3B+Grewal%2C+R.+et+al.&rft.au=Abayomi%2C+A.%3B+Christoffels%2C+A.%3B+Grewal%2C+R.+et+al.&rft.date=2013&rft.volume=11&rft.issue=6&rft.pages=347%E2%80%93354&rft_id=info:doi\/10.1089%2Fbio.2013.0049&rft_id=info:pmid\/24835364&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DhaiEstab13-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DhaiEstab13_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dhai, A. (2013). \"Establishing national biobanks in South Africa: The urgent need for an ethico-regulatory framework\". <i>The South African Journal of Bioethics & Law<\/i> <b>6<\/b> (2): 38\u201339. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7196%2FSAJBL.296\" target=\"_blank\">10.7196\/SAJBL.296<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Establishing+national+biobanks+in+South+Africa%3A+The+urgent+need+for+an+ethico-regulatory+framework&rft.jtitle=The+South+African+Journal+of+Bioethics+%26+Law&rft.aulast=Dhai%2C+A.&rft.au=Dhai%2C+A.&rft.date=2013&rft.volume=6&rft.issue=2&rft.pages=38%E2%80%9339&rft_id=info:doi\/10.7196%2FSAJBL.296&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-deVriesAPerpetual14-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-deVriesAPerpetual14_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">de Vries, J.; Abayomi, A.; Brandful, J. et al. (2014). \"A perpetual source of DNA or something really different: ethical issues in the creation of cell lines for African genomics research\". <i>BMC Medical Ethics<\/i> <b>15<\/b>: 60. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6939-15-60\" target=\"_blank\">10.1186\/1472-6939-15-60<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+perpetual+source+of+DNA+or+something+really+different%3A+ethical+issues+in+the+creation+of+cell+lines+for+African+genomics+research&rft.jtitle=BMC+Medical+Ethics&rft.aulast=de+Vries%2C+J.%3B+Abayomi%2C+A.%3B+Brandful%2C+J.+et+al.&rft.au=de+Vries%2C+J.%3B+Abayomi%2C+A.%3B+Brandful%2C+J.+et+al.&rft.date=2014&rft.volume=15&rft.pages=60&rft_id=info:doi\/10.1186%2F1472-6939-15-60&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BIKA-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BIKA_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bikalims.org\/\" target=\"_blank\">\"BIKA\"<\/a>. Bika Lab Systems (Pty) Ltd<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bikalims.org\/\" target=\"_blank\">https:\/\/www.bikalims.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 16 August 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=BIKA&rft.atitle=&rft.pub=Bika+Lab+Systems+%28Pty%29+Ltd&rft_id=https%3A%2F%2Fwww.bikalims.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OpenSpecimen-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OpenSpecimen_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.openspecimen.org\/\" target=\"_blank\">\"OpenSpecimen\"<\/a>. Krishagni Solutions Pvt. Ltd<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.openspecimen.org\/\" target=\"_blank\">http:\/\/www.openspecimen.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 16 August 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OpenSpecimen&rft.atitle=&rft.pub=Krishagni+Solutions+Pvt.+Ltd&rft_id=http%3A%2F%2Fwww.openspecimen.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NCB-H3ACape13-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NCB-H3ACape13_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">NCB-H3A (23 March 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/christoffels.sanbi.ac.za\/index.php\/projects\/biobanking\" target=\"_blank\">\"NCB-H3A Cape Town Biobank: Biobank Information Management System - Functional Requirements Overview & Phase I Objectives\"<\/a>. South African National Bioinformatics Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/christoffels.sanbi.ac.za\/index.php\/projects\/biobanking\" target=\"_blank\">http:\/\/christoffels.sanbi.ac.za\/index.php\/projects\/biobanking<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 16 August 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NCB-H3A+Cape+Town+Biobank%3A+Biobank+Information+Management+System+-+Functional+Requirements+Overview+%26+Phase+I+Objectives&rft.atitle=&rft.aulast=NCB-H3A&rft.au=NCB-H3A&rft.date=23+March+2013&rft.pub=South+African+National+Bioinformatics+Institute&rft_id=http%3A%2F%2Fchristoffels.sanbi.ac.za%2Findex.php%2Fprojects%2Fbiobanking&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DowstAcquire15-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DowstAcquire15_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-DowstAcquire15_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dowst, H.; Pew, B.; Watkins, C. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4426840\" target=\"_blank\">\"Acquire: An open-source comprehensive cancer biobanking system\"<\/a>. <i>Bioinformatics<\/i> <b>31<\/b> (10): 1655\u201362. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtv012\" target=\"_blank\">10.1093\/bioinformatics\/btv012<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4426840\/\" target=\"_blank\">PMC4426840<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25573920\" target=\"_blank\">25573920<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4426840\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4426840<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Acquire%3A+An+open-source+comprehensive+cancer+biobanking+system&rft.jtitle=Bioinformatics&rft.aulast=Dowst%2C+H.%3B+Pew%2C+B.%3B+Watkins%2C+C.+et+al.&rft.au=Dowst%2C+H.%3B+Pew%2C+B.%3B+Watkins%2C+C.+et+al.&rft.date=2015&rft.volume=31&rft.issue=10&rft.pages=1655%E2%80%9362&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtv012&rft_id=info:pmc\/PMC4426840&rft_id=info:pmid\/25573920&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4426840&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Fearn2012_13-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Fearn2012_13_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fearn, P.; Michels, C.; Meagher, K.; Cada, M. (2013). \"2012 International Society for Biological and Environmental Repositories Informatics Working Group: Survey results and conclusions\". <i>Biopreservation and Biobanking<\/i> <b>11<\/b> (1): 64-6. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fbio.2012.1115\" target=\"_blank\">10.1089\/bio.2012.1115<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24845257\" target=\"_blank\">24845257<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=2012+International+Society+for+Biological+and+Environmental+Repositories+Informatics+Working+Group%3A+Survey+results+and+conclusions&rft.jtitle=Biopreservation+and+Biobanking&rft.aulast=Fearn%2C+P.%3B+Michels%2C+C.%3B+Meagher%2C+K.%3B+Cada%2C+M.&rft.au=Fearn%2C+P.%3B+Michels%2C+C.%3B+Meagher%2C+K.%3B+Cada%2C+M.&rft.date=2013&rft.volume=11&rft.issue=1&rft.pages=64-6&rft_id=info:doi\/10.1089%2Fbio.2012.1115&rft_id=info:pmid\/24845257&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NHSEnglandQICExec13-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NHSEnglandQICExec13_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Quality Information Committee (April 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.england.nhs.uk\/wp-content\/uploads\/2013\/04\/1ndqr-exec-sum.pdf\" target=\"_blank\">\"First National Data Quality Review: Executive Summary\"<\/a> (PDF). NHS England<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.england.nhs.uk\/wp-content\/uploads\/2013\/04\/1ndqr-exec-sum.pdf\" target=\"_blank\">https:\/\/www.england.nhs.uk\/wp-content\/uploads\/2013\/04\/1ndqr-exec-sum.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 August 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=First+National+Data+Quality+Review%3A+Executive+Summary&rft.atitle=&rft.aulast=Quality+Information+Committee&rft.au=Quality+Information+Committee&rft.date=April+2013&rft.pub=NHS+England&rft_id=https%3A%2F%2Fwww.england.nhs.uk%2Fwp-content%2Fuploads%2F2013%2F04%2F1ndqr-exec-sum.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Merino-MartinezrToward16-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Merino-MartinezrToward16_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Merino-Martinez, R.; Norlin, L.; van Enckevort, D. et al. (2016). \"Toward Global Biobank Integration by Implementation of the Minimum Information About BIobank Data Sharing (MIABIS 2.0 Core)\". <i>Biopreservation and Biobanking<\/i> <b>14<\/b> (4): 298-306. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fbio.2015.0070\" target=\"_blank\">10.1089\/bio.2015.0070<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26977825\" target=\"_blank\">26977825<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+Global+Biobank+Integration+by+Implementation+of+the+Minimum+Information+About+BIobank+Data+Sharing+%28MIABIS+2.0+Core%29&rft.jtitle=Biopreservation+and+Biobanking&rft.aulast=Merino-Martinez%2C+R.%3B+Norlin%2C+L.%3B+van+Enckevort%2C+D.+et+al.&rft.au=Merino-Martinez%2C+R.%3B+Norlin%2C+L.%3B+van+Enckevort%2C+D.+et+al.&rft.date=2016&rft.volume=14&rft.issue=4&rft.pages=298-306&rft_id=info:doi\/10.1089%2Fbio.2015.0070&rft_id=info:pmid\/26977825&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BCCTBSearchArch-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BCCTBSearchArch_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20140801214940\/https:\/\/breastcancertissuebank.org\/bcc\/tissueBank?Name=sampleFinder\" target=\"_blank\">\"Breast Cancer Campaign Tissue Bank - Search\"<\/a>. Breast Cancer Campaign. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/breastcancertissuebank.org\/bcc\/tissueBank?Name=sampleFinder\" target=\"_blank\">the original<\/a> on 01 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20140801214940\/https:\/\/breastcancertissuebank.org\/bcc\/tissueBank?Name=sampleFinder\" target=\"_blank\">https:\/\/web.archive.org\/web\/20140801214940\/https:\/\/breastcancertissuebank.org\/bcc\/tissueBank?Name=sampleFinder<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 August 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Breast+Cancer+Campaign+Tissue+Bank+-+Search&rft.atitle=&rft.pub=Breast+Cancer+Campaign&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20140801214940%2Fhttps%3A%2F%2Fbreastcancertissuebank.org%2Fbcc%2FtissueBank%3FName%3DsampleFinder&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-QuinlanTheInfo15-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-QuinlanTheInfo15_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-QuinlanTheInfo15_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Quinlan, P.R.; Groves, M.; Jordan, L.B. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4675179\" target=\"_blank\">\"The Informatics Challenges Facing Biobanks: A Perspective from a United Kingdom Biobanking Network\"<\/a>. <i>Biopreservation and Biobanking<\/i> <b>13<\/b> (5): 363-70. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fbio.2014.0099\" target=\"_blank\">10.1089\/bio.2014.0099<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4675179\/\" target=\"_blank\">PMC4675179<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26418270\" target=\"_blank\">26418270<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4675179\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4675179<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Informatics+Challenges+Facing+Biobanks%3A+A+Perspective+from+a+United+Kingdom+Biobanking+Network&rft.jtitle=Biopreservation+and+Biobanking&rft.aulast=Quinlan%2C+P.R.%3B+Groves%2C+M.%3B+Jordan%2C+L.B.+et+al.&rft.au=Quinlan%2C+P.R.%3B+Groves%2C+M.%3B+Jordan%2C+L.B.+et+al.&rft.date=2015&rft.volume=13&rft.issue=5&rft.pages=363-70&rft_id=info:doi\/10.1089%2Fbio.2014.0099&rft_id=info:pmc\/PMC4675179&rft_id=info:pmid\/26418270&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4675179&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-M.C3.BCllerFrom17-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-M.C3.BCllerFrom17_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">M\u00fcller, H.; Malservet, N.; Quinlan, P. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5346419\" target=\"_blank\">\"From the evaluation of existing solutions to an all-inclusive package for biobanks\"<\/a>. <i>Health and Technology<\/i> <b>7<\/b> (1): 89-95. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs12553-016-0175-x\" target=\"_blank\">10.1007\/s12553-016-0175-x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5346419\/\" target=\"_blank\">PMC5346419<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28344915\" target=\"_blank\">28344915<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5346419\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5346419<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+the+evaluation+of+existing+solutions+to+an+all-inclusive+package+for+biobanks&rft.jtitle=Health+and+Technology&rft.aulast=M%C3%BCller%2C+H.%3B+Malservet%2C+N.%3B+Quinlan%2C+P.+et+al.&rft.au=M%C3%BCller%2C+H.%3B+Malservet%2C+N.%3B+Quinlan%2C+P.+et+al.&rft.date=2017&rft.volume=7&rft.issue=1&rft.pages=89-95&rft_id=info:doi\/10.1007%2Fs12553-016-0175-x&rft_id=info:pmc\/PMC5346419&rft_id=info:pmid\/28344915&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5346419&rfr_id=info:sid\/en.wikipedia.org:Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The link to the Breast Cancer Campaign Tissue Bank search (reference 12) is broken; an archived version of the link is used instead.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181457\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.420 seconds\nReal time usage: 0.442 seconds\nPreprocessor visited node count: 11965\/1000000\nPreprocessor generated node count: 30971\/1000000\nPost\u2010expand include size: 96295\/2097152 bytes\nTemplate argument size: 33159\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 418.904 1 - -total\n 76.36% 319.893 1 - Template:Reflist\n 66.57% 278.861 14 - Template:Citation\/core\n 52.22% 218.765 9 - Template:Cite_journal\n 18.57% 77.811 1 - Template:Infobox_journal_article\n 18.48% 77.427 5 - Template:Cite_web\n 17.89% 74.939 1 - Template:Infobox\n 11.04% 46.245 80 - Template:Infobox\/row\n 8.07% 33.785 19 - Template:Citation\/identifier\n 3.65% 15.271 15 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10120-0!*!0!!en!5!* and timestamp 20181214181456 and revision id 30541\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking\">https:\/\/www.limswiki.org\/index.php\/Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","9e160b0ad749c44fbc6303a7527b6e1e_images":["https:\/\/www.limswiki.org\/images\/7\/7a\/Fig1_Bendou_BiopresAndBiobank2017_15-2.gif"],"9e160b0ad749c44fbc6303a7527b6e1e_timestamp":1544811296,"a67695148af73db7d41c4171d7c86cbc_type":"article","a67695148af73db7d41c4171d7c86cbc_title":"Effective information extraction framework for heterogeneous clinical reports using online machine learning and controlled vocabularies (Zheng et al. 2017)","a67695148af73db7d41c4171d7c86cbc_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies","a67695148af73db7d41c4171d7c86cbc_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Effective information extraction framework for heterogeneous clinical reports using online machine learning and controlled vocabularies\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nEffective information extraction framework for heterogeneous clinical reports using online machine learning and controlled vocabulariesJournal\n \nJMIR Medical InformaticsAuthor(s)\n \nZheng, Shuai; Lu, James. J.; Ghasemzadeh, Nima; Hayek, Salim S.; Quyyumi, Arshed, A.; Wang, FushengAuthor affiliation(s)\n \nEmory University, Stony Brook UniversityPrimary contact\n \nEmail: fusheng dot wang at stonybrook dot edu; Phone: 1 6316327528Editors\n \nEysenbach, G.Year published\n \n2017Volume and issue\n \n5 (2)Page(s)\n \ne12DOI\n \n10.2196\/medinform.7235ISSN\n \n2291-9694Distribution license\n \nCreative Commons Attribution 2.0Website\n \nhttp:\/\/medinform.jmir.org\/2017\/2\/e12\/Download\n \nhttp:\/\/medinform.jmir.org\/2017\/2\/e12\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Background \n\n3.1 Related work \n3.2 Heterogeneous clinical reports \n\n\n4 Methods \n\n4.1 Overview \n\n4.1.1 System interface and user operations \n4.1.2 Document preprocessing \n4.1.3 Answer prediction \n4.1.4 Learning \n\n\n\n\n5 Results \n\n5.1 Experimental setup \n\n5.1.1 Datasets \n5.1.2 Ground truth \n5.1.3 Evaluation metrics \n5.1.4 Experiment settings \n\n\n\n\n6 Discussion \n\n6.1 Principal findings \n6.2 Conclusions \n\n\n7 Acknowledgments \n8 Conflicts of interest \n9 Multimedia Appendix 1 \n10 References \n11 Abbreviations \n12 Notes \n\n\n\nAbstract \nBackground: Extracting structured data from narrated medical reports is challenged by the complexity of heterogeneous structures and vocabularies and often requires significant manual effort. Traditional machine-based approaches lack the capability to take user feedback for improving the extraction algorithm in real time.\nObjective: Our goal was to provide a generic information extraction framework that can support diverse clinical reports and enables a dynamic interaction between a human and a machine that produces highly accurate results.\nMethods: A clinical information extraction system IDEAL-X has been built on top of online machine learning. It processes one document at a time, and user interactions are recorded as feedback to update the learning model in real time. The updated model is used to predict values for extraction in subsequent documents. Once prediction accuracy reaches a user-acceptable threshold, the remaining documents may be batch processed. A customizable controlled vocabulary may be used to support extraction.\nResults: Three datasets were used for experiments based on report styles: 100 cardiac catheterization procedure reports, 100 coronary angiographic reports, and 100 integrated reports \u2014 each combines history and physical report, discharge summary, outpatient clinic notes, outpatient clinic letter, and inpatient discharge medication report. Data extraction was performed by three methods: online machine learning, controlled vocabularies, and a combination of these. The system delivers results with F1 scores greater than 95%.\nConclusions: IDEAL-X adopts a unique online machine learning\u2013based approach combined with controlled vocabularies to support data extraction for clinical reports. The system can quickly learn and improve, thus it is highly adaptable.\nKeywords: information extraction, natural language processing, controlled vocabulary, electronic medical records \n\nIntroduction \nWhile immense efforts have been made to enable a structured data model for electronic medical records (EMRs), a large amount of medical data remain in free-form narrative text, and useful data from individual patients are usually distributed across multiple reports of heterogeneous structures and vocabularies. This poses major challenges to traditional information extraction systems, as either costly training datasets or manually crafted rules have to be prepared. These approaches also lack the capability of taking user feedback to adapt and improve the extraction algorithm in real time.\nOur goal is to provide a generic information extraction framework that adapts to diverse clinical reports, enables a dynamic interaction between a human and a machine, and produces highly accurate results with minimal human effort. We have developed a system, Information and Data Extraction using Adaptive Online Learning (IDEAL-X), to support adaptive information extraction from diverse clinical reports with heterogeneous structures and vocabularies. The system is built on top of online machine learning and customizable controlled vocabularies. A demo video can be found on YouTube.[1]\nIDEAL-X uses an online machine learning\u2013based approach[2][3][4] for information extraction. Traditional machine learning algorithms take a two-stage approach: batch training based on an annotated training dataset, and batch prediction for future datasets based on the model generated from stage one (Figure 1). In contrast, online machine learning algorithms[2][3] take an iterative approach (Figure 1). IDEAL-X learns one document at a time and predicts values to be extracted for the next one. Learning occurs from revisions made by the user, and the updated model is applied to prediction for subsequent documents. Once the model achieves a satisfactory accuracy, the remaining documents may be processed in batches. Online machine learning not only significantly reduces user effort for annotation but also provides the mechanism for collecting feedback from human-machine interaction to improve the system's model continuously.\nBesides online machine learning, IDEAL-X allows for customizable controlled vocabularies to support data extraction from clinical reports, where a vocabulary enumerates the possible values that can be extracted for a given attribute. (The X in IDEAL-X represents the controlled vocabulary plug-in.) The use of online machine learning and controlled vocabularies is not mutually exclusive; they are complementary, which provides the user with a variety of modes for working with IDEAL-X.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 Online machine learning versus batch learning. (a) Batch machine learning workflow; (b) Online machine learning workflow\n\n\n\nBackground \nRelated work \nA number of research efforts have been made in different fields of medical information extraction. Successful systems include caTIES[5], MedEx\n[6], MedLEE[7], cTAKES[8], MetaMap[9], HITEx[10], and so on. These methods either take a rule-based approach, a traditional machine learning\u2013based approach, or a combination of both.\nDifferent online learning algorithms have been studied and developed for classification tasks[11], but their direct application to information extraction has not been studied. Especially in the clinical environment, the effectiveness of these algorithms is yet to be examined. Several pioneering projects have used learning processes that involve user interaction and certain elements of IDEAL-X. I2E2 is an early rule-based interactive information extraction system.[12] It is limited by its restriction to a predefined feature set. Amilcare[13][14] is adaptable to different domains. Each domain requires an initial training that can be retrained on the basis of the user\u2019s revision. Its algorithm (LP)2 is able to generalize and induce symbolic rules. RapTAT[15] is most similar to IDEAL-X in its goals. It preannotates text interactively to accelerate the annotation process. It uses a multinominal na\u00efve Baysian algorithm for classification but does not appear to use contextual information beyond previously found values in its search process. This may limit its ability to extract certain value types.\nDifferent from online machine learning but related is active learning[16][17]; it assumes the ability to retrieve labels for the most informative data points while involving the users in the annotation process. DUALIST[18] allows users to select system-populated rules for feature annotation to support information extraction. Other example applications in health care informatics include word sense disambiguation[19] and phenotyping.[20] Active learning usually requires comprehending the entire corpus in order to pick the most useful data point. However, in a clinical environment, data arrive in a streaming fashion over time, which limits our ability to choose data points. Hence, an online learning approach is more suitable.\nIDEAL-X adopts the Hidden Markov Model for its compatibility with online learning, and for its efficiency and scalability. We will also describe a broader set of contextual information used by the learning algorithm to facilitate extraction of values of all types.\n\nHeterogeneous clinical reports \nA patient\u2019s electronic medical record could come with a variety of medical reports. Data in these reports provide critical information that can be used to improve clinical diagnosis and support biomedical research. For example, the Emory University Cardiovascular Biobank[21] collects records of patients with potential or confirmed coronary artery diseases undergoing cardiac catheterization, and it aims to combine extracted data elements from multiple reports to identity patients for research. Report types include history and physical report, discharge summary, outpatient clinic note, outpatient clinic letter, coronary angiogram report, cardiac catheterization procedure report, echocardiogram report, inpatient report, and discharge medication lists.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2 Example snippets of different report forms. (a) Semi-structured report; (b) Template based narration; and (c) Complex narration\n\n\n\nWe classify clinical reports into three forms: semi-structured data, template-based narration, and complex narration. Semi-structured data represent data elements in the form of attribute and value pairs (Figure 2). Reports in this form have simple structures, making data extraction relatively straightforward. Template-based narration is a very common report form. The narrative style, including sentence patterns and vocabularies, follows consistent templates and expressions (Figure 2). Extracting information from this type of text (e.g., \"right posterior descending artery\") require major linguistics expertise, to either formulate extraction rules or to annotate training data. Complex narration is essentially free-form text. It can be irregular, personal, and idiomatic (Figure 2). Most medical reporting systems still allow for (and thus encourage) such a style. It is the most difficult form to interpret and process by natural language processing (NLP) algorithms. Nevertheless, certain types of information such as diseases and medications have finite vocabulary that could be used to support data extraction.\n\nMethods \nOverview \nThe interface and workflow conform to traditional annotation systems: a user browses an input document from the input document collection and fills out an output form. On loading each document, the system attempts to fill the output form automatically with its data extraction engine. Then, a user can review and revise incorrect answers. The system then updates its data extraction model automatically based on the user\u2019s feedback. Optionally, the user may provide a customized controlled vocabulary to further support data extraction and answer normalization. Pretraining with manually annotated data is not required, as the prediction model behind the data extraction engine can be established incrementally through online learning, customizing controlled vocabularies, or a combination of the two.\nThe system can operate in two modes: (1) interactive: through online learning, the system predicts values to be extracted for each report, and the user verifies or corrects the predicted values; and (2) batch: batch predicting for all unprocessed documents once the accrued accuracy is sufficient for users. Whereas interactive mode uses online machine learning to build the learning model incrementally, batch mode runs the same as the prediction phase of batch machine learning.\n\nSystem interface and user operations \nSystem interface\nIDEAL-X provides a GUI with two main panels, a menu, and navigation buttons (Figure 3). The left panel is for browsing an input report, and the right panel is the output table with predicted values of each data element in the report. The menu provides options for defining the data elements to be extracted, specifying input reports, among others.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 3 An example screenshot of IDEAL-X\u2019s interface\n\n\n\nDefinition of data elements for extraction\nThe system provides a wizard for constructing the metadata of the output form. The user builds the form by specifying a list of data elements and their constraints. An example is the data element \"Heart Rate,\" which is constrained to be a numerical value between 0 and 200. Other constraints include sections of the report that may contain the values. However, except for the names of the data elements, specifying constraints are optional, as these can be learned by the system.\n\nData extraction workflow\nThe user will first select a collection of input reports to be extracted from a local folder. By default, the system runs in an interactive mode, and one report will be loaded at a time on the left display panel. The user can make manual annotations by highlighting the correct value in the report text. Clicking the corresponding data field in the table assigns the value to the data element. If the system has prefilled the field of a data element with a predicted value, the user can provide feedback by fixing incorrect values. As the user navigates to the next document, the system compares the prefilled and the final values for the most recently processed document. Values that are unchanged or filled in by users are taken as positive instances, and values that have been revised are taken as negative instances. Both instances are incorporated into the online learning algorithm to be used by the data extraction for subsequent documents. By iterating through this process, the amount of information that the system is able to correctly prefill grows over time. Note that manual revision in this context is different from traditional human labeling. It is only necessary if there is a wrong prediction, thus user effort can be significantly saved. Once the decision model reaches an acceptable level of accuracy, the user has the option to switch to batch mode to complete extraction for the remaining documents. If a patient has multiple reports, the text input panel displays each report with a separate tab. Data extracted from all the reports are aggregated in the output.\n\nCustomization of controlled vocabularies\nIDEAL-X also provides an interface for the user to customize a controlled vocabulary that can be used by the system for data extraction. The controlled vocabulary contains both terminology and structural properties. The terminology includes lists of values and their normalization mappings. For example, disease terminology includes \"Diabetes Mellitus\" with variations \"DM\" and \"Diabetes.\" It also defines inductions. For example, taking \"Insulin\" or \"Metformin\" indicates having diabetes mellitus. Structural properties provide positive and negative contextual information for giving terms. For example, to extract medications taken by patients, the \"Allergies\" section is a negative context and medicine names in the section will be skipped. Structural properties may also contain disambiguation terms that may further improve the precision of extraction. A simple example is that \"intolerant\" is a negative indicator for identifying \"statin\" as \"statin intolerant\" refers to a different concept. Controlled vocabularies can be a powerful tool to support data extraction: it can be used to locate sentences and chunks of possible values, and to perform normalization for extracted values, discussed in the next section.\nThe data extraction engine\nWhile the user interacts with IDEAL-X interface, the data extraction engine works transparently in the background. The engine has three major components: answer prediction, learning, and the learning model that the online learning process continuously updates (Figure 4). The system combines statistical and machine learning\u2013based approaches with controlled vocabularies for effective data extraction.\n\nDocument preprocessing \nWhen a report is loaded, the text is first parsed into an in-memory hierarchical tree consisting of four layers: section, paragraph, sentence, and token. Apache OpenNLP[22] is used to support the parsing with its Sentence Detector, Tokenizer, and Part-of-Speech Tagger. A reverse index of tokens is created to support an efficient keyword-based search. The index is used to find locations (e.g., sections, paragraphs, sentences, and phrases) of a token, as well as its properties such as part of speech and data type. For example, given the token \"DM,\" the system can quickly identify the section (e.g., \"History\") and the containing sentences. Such token search is frequently performed in answer prediction, and the in-memory index structures enable high efficiency for such operations.\n\nAnswer prediction \nPredicting the value of each data element involves the following steps: (1) Identifying target sentences that are likely to contain the answer; (2) Identifying candidate chunks in the sentences; (3) Filtering the chunks to generate candidate values; (4) Ranking candidate values to generate (raw) values; (5) Normalizing values; and (6) Aggregating values from multiple reports. The workflow is shown in Figure 4.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 4 Overview of System Workflow\n\n\n\nIdentifying target sentences\nThrough online learning, the system accrues keywords from past answers (answer keywords) along with co-occurring words in the corresponding sentences (contextual words). For example, given the answer keywords \"diabetes\" and \"hypertension\" in the sentence \"The patient reports history of diabetes and hypertension,\" contextual words are \"patient,\" \"report,\" and \"history.\" Such answer keywords and contextual words combined with customized vocabularies can be utilized to identify sentences that are likely to contain answers with the following methods:\nFirst, there's similarity-based search using the vector space model.[23] Given a collection of contextual words and their frequencies, the system computes the similarity against sentences in the document.[23] Sentences with high similarities are selected. For example, most sentences about \"disease\" contain \"diagnosis\" and \"history.\" The past contextual keywords and their frequency weights are represented and maintained through a learning model discussed later in section on Learning.\nSecond, we have the answer keyword matching search. The answer keywords, combined with relevant user customized vocabularies, are also used to identify target sentences with keyword matching. For example, to extract diseases, if a sentence contains the disease term \"myocardial infarction\" defined in the vocabulary, the sentence is selected as a target. In both approaches, sections to be searched or skipped are also considered to narrow the scope of searching.\nIdentifying candidate chunks\nAfter target sentences are selected, the system identifies potential phrases in the sentences using two methods: Hidden Markov model (HMM)[24] and keyword-based search. The HMM represents target words and contextual words in a sentence with different states, and it marks values to be extracted based on probability distributions learned from previously collected values and their sentences. The keyword-based search finds candidate chunks using keywords collected from past answers and the controlled vocabulary.\n\nFiltering chunks\nTo filter candidate chunks, the system uses rule induction[14][25] to generate \"If-Then\" rules based on historical statistics. The following filtering criteria are used: (1) Part of speech (POS): This filters a phrase by its POS tag in the sentence. Simple example phrases are noun and verb phrases. (2) String pattern: This looks for chunks that match special string patterns. For example, the first characters of all tokens are capitalized. (3) Value domain: This eliminates numerical or enumerated values that fall outside a specified range of values. (4) Negation: Based on predefined built-in rules, this removes phrases governed by words that reverse the meaning of the answer.[26] For example, if a candidate chunk \"cancer\" is extracted from a sentence \"the patient has no history of cancer,\" \"cancer\" would not be included. (5) Certainty: Similar to negation filter, this detects and filters uncertain events or situations such as future plans, based on predefined rules. For example, a candidate chunk \"radiation therapy\" for treatment from a sentence \"the patient is planned to take radiation therapy\" should not be included. Whereas negation and certainty filtering is based on predefined rules, other filtering relies on real-time data statistics for filtering criteria.\n\nRanking candidate values\nThe system combines the scores of the selected sentences and chunks for ranking of candidate values. For a single-valued data element (e.g., heart beat), the candidate value with the highest confidence score is selected. For a multi-valued data element (e.g., medication), values with confidence scores above a threshold are selected. Based on this, each candidate value is either accepted or rejected.\nNormalizing values\nThis step normalizes extracted values through transformation, generalization, and induction rules given by the controlled vocabulary (Figure 4). For example, \"DM\" is transformed into \"Diabetes Mellitus.\" \"Pindolol\" is generalized to its hypernym \"beta blocker.\" The appearance of the medication term \"Metformin\" (a drug for treating type 2 diabetes) in the text can infer the disease \"Diabetes Mellitus.\"\nAggregating results\nData extracted from multiple reports of a patient will be aggregated into a single table. The aggregation process may normalize values and remove duplicates. For example, \"lisinopril\" and \"captopril\" are extracted from discharge summary and inpatient report, respectively, and they can be normalized as \"ACE inhibitor.\" If the same data element is extracted from multiple reports, deduplication is performed. The final output is in simple structural table form that can be exported conveniently to other applications such as Excel (Microsoft) or a database.\nNote that controlled vocabularies can play important roles in the answer prediction process. They are used for identifying target sentences through keyword searching, identifying candidate chunks through keyword matching, and supporting normalization for extracted values.\n\nLearning \nDEAL-X takes an online learning\u2013based approach to incrementally build statistical models and make predictions (Figure 5). The three models used in IDEAL-X are all statistical based and can be continuously updated after each iteration.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 5 Precision and recall changes over processed records\n\n\n\nSystem-predicted values automatically populate the output table, and the user advances to the next report with or without revision to these values. In both cases, the internal learning and prediction models of IDEAL-X are updated. For each instance, IDEAL-X collects and analyzes the following features: (1) Position: location of the answer in the text hierarchy; (2) Landmark: co-occurring contextual keywords in a sentence; (3) POS: parts of speech tag; (4) Value: the tokens of the answer; and (5) String patterns: literal features such as capitalization and initial and special punctuation. These features are then used to update the three models.\nIn IDEAL-X, each data element such as attribute \"disease\" or \"medicine\" has its own statistical model, and each new instance of a data element will update the corresponding model. There are three models to be updated: (1) Updating Space Vector Model: This model uses \"Landmark\" features of positive instances. The system updates frequencies of co-occurring contextual words, used as weights of the space vector.[23] (2) Updating HMM: HMM lists all words in a sentence as a sequence, in which an extracted value is marked as target value state and other words are recognized as irrelevant contextual states. Based on this sequence, the state transition probabilities and emission probabilities are recalculated.[24] (3) Updating rule induction model: Filtering rules are induced based on the coverage percentage.[25] Features such as POS, value domain, and string patterns of both positive and negative instances are analyzed and their respective coverage percentages are modified. Once the coverage of a rule reaches a predefined threshold, the rule is triggered for filtering.\nIn an interactive mode, the above four steps repeat for each report, where the learning models are continuously updated and improved.\n\nResults \nExperimental setup \nDatasets \nWe used three datasets from 100 patients that were randomly sampled from a collection of about 5000 patients in the Emory Biobank database. Dataset 1 is a set of semi-structured reports and contains 100 cardiac catheterization procedure reports. Dataset 2 is a set of template-based narration and contains 100 coronary angiographic reports. Dataset 3 is a set of complex narration and contains 315 reports, including history and physical report, discharge summary, outpatient clinic notes, outpatient clinic letter, and inpatient discharge medication report.\n\nGround truth \nThe test datasets are independently hand-annotated by domain expert annotators, including physicians, physician trainees, and students trained by the Emory Clinical Cardiovascular Research Institute for Biobank data reporting. Each record is annotated by two different annotators. The interrater agreement scores (kappa) of these three datasets are .991, .986, and .835, respectively. An arbitrator \u2014 an independent cardiovascular disease researcher \u2014 reconciles incompatible outputs of the system and the manual annotations to produce the final ground truth.\n\nEvaluation metrics \nFor validation, precision, recall, and F1, scores are used to estimate the effectiveness of extraction by comparing the system predicted results (before human revision) and the ground truth.\n\nExperiment settings \nWe aimed to evaluate the effectiveness of the system with respect to using online learning and controlled vocabularies and to understand their applicability to different report forms. By analyzing the report styles and vocabularies, we discovered that online learning will be more suitable for semi-structured or template-based narration reports, and controlled vocabulary-guided data extraction would be more effective on complex narration with a finite vocabulary. Thus, we designed three experiments: (1) Online learning\u2013based data extraction, where controlled vocabularies are not provided, based on Dataset 1 (semi-structured) and Dataset 2 (template-based narration); (2) Controlled vocabularies-based data extraction, where online learning is not used, based on Dataset 3 (complex narration); and (3) Controlled vocabularies-guided data extraction combined with online learning, based on Dataset 3.\nPerformance evaluation\nExperiment 1: Online machine learning\u2013based data extraction: This experiment was based on Datasets 1 and 2. The system starts in an interactive mode with an empty decision model without prior training. The defined data elements are summarized in Multimedia Appendix 1. The user processes one report at a time, and each system-predicted value (including empty values for the first few reports) before user revision was recorded for calculating precision and recall.\nResults are summarized in Table 1 for the two datasets, respectively. Both test cases achieved high precision as semi-structured and template-based text is most easy to handle. To study the learning rate of online learning, we divided records into 10 groups and plotted precision and recall of every 10 percent of the records in Datasets 1 and 2. We observed that in both tests, the system maintained high precision during the learning process. Although some variability exists due to new data pattern, the recall of both cases also improved steadily. Not surprisingly, the rate of learning for Dataset 1 is much faster given its semi-structure.\n\n\n\n\n\n\n\nTable 1. Results of data extraction from semistructured reports (Dataset 1) and template-based narration (Dataset 2)\n\n\nDataset\n\nNumber of data elements\n\nNumber of values\n\nPrecision (%)\n\nRecall (%)\n\nF1 (%)\n\n\n1\n\n19\n\n1272\n\n99.8\n\n96.5\n\n98.1\n\n\n2\n\n16\n\n728\n\n97.2\n\n93.2\n\n95.2\n\n\n\nExperiment 2: Controlled vocabularies-guided data extraction: In this experiment, online learning was disabled and data extraction was performed in batches using controlled vocabulary. Diseases and medications were extracted from Dataset 3 (values to be extracted are shown in Multimedia Appendix 1). Customized controlled vocabularies, including terminology and structural properties, had been created independently by physicians via references to domain knowledge resources and analyzing another development report dataset of 100 patients, disjointed from Dataset 3. Note that comparisons in this and the following experiments were at a clinical finding level between system-integrated results and manual-annotation-integrated results.\nThe results in Table 2 show that controlled vocabularies are highly effective for data extraction over complex narratives. Domain-specific data such as cardiology-related diseases and medications have limited numbers of possible values (or domain values), and a carefully customized controlled vocabulary can achieve high extraction accuracy.\n\n\n\n\n\n\n\nTable 2. Results of controlled vocabularies-guided data extraction from complex narration (Dataset 3)\n\n\nType of data elements\n\nNumber of data elements\n\nNumber of round truth values\n\nPrecision (%)\n\nRecall (%)\n\nF1 (%)\n\n\nDiseases\n\n15\n\n418\n\n94.5\n\n99.0\n\n96.7\n\n\nMedications\n\n10\n\n437\n\n98.6\n\n99.7\n\n99.2\n\n\nAll\n\n25\n\n855\n\n96.5\n\n99.4\n\n97.9\n\n\n\nExperiment 3: Controlled vocabularies-guided data extraction combined with online machine learning: In this experiment, we performed two tests to examine how efficient and effective the system learns when only terminology is available and structural properties need to be obtained from online learning. Test 1 was to generate the baseline for comparison, and Test 2 was to demonstrate the effectiveness of combining online machine learning and controlled vocabularies. Dataset 3 was used to extract all diseases and medications.\nFor Test 1, terminology was used and online machine learning was disabled, so the test was guided by controlled vocabulary without any structural properties. We note that comprehensive terminology contributes directly to high recall rate, which means that the system seldom misses values to be extracted. However, if structural properties are not included, compared with the result in Experiment 2, the precision is much lower. This highlights the value of positive and negative contexts in an extraction task.\nFor Test 2, both terminology and online machine learning were used. Online machine learning supports learning structural properties. To show how quickly the system learns, only the 38 reports associated with the first 10 patients were processed with interactive online learning. All remaining reports were processed in batch. Results in Table 3 show an overall precision of 94.9%, which demonstrates that online learning could quickly learn structural properties.\n\n\n\n\n\n\n\nTable 3. Results of controlled vocabularies-guided data extraction combined with online learning\n\n\nTest\n\nControlled vocabulary\n\nOnline learning\n\nPrecision (%)\n\nRecall (%)\n\nF1 (%)\n\n\n1\n\nTerminology only\n\nN\/A\n\n80.9\n\n99.4\n\n89.2\n\n\n2\n\nTerminology only\n\nApplied to first 10 patients\n\n94.9\n\n99.4\n\n97.1\n\n\n\nTypical errors in these two tests were associated with terminology and contextual information used in complex narrative scenarios. On one hand, the completeness of the terminology list, including terms and their synonyms, influences the recall rate directly. On the other hand, although coverage of terminologies could be maximized by a carefully engineered vocabulary, unwanted extractions arising from searches in the wrong section, undetected negations, and ambiguous use of terms can still lower the overall precision.\n\nDiscussion \nPrincipal findings \nIDEAL-X provides a generic data extraction framework that takes advantage of both online learning and controlled vocabularies. The two approaches complement each other and can also be combined. An online learning\u2013based approach is highly effective for reports with underlying structural patterns such as semi-structured or template-based narration style-based reports. Experiments with complex narrative reports indicate that the use of controlled vocabularies is highly effective for supporting extraction constrained by a finite data domain. In addition, structural properties such as section-data associations can play an important role in improving the accuracy of extraction. However, in cases where controlled vocabularies are unavailable \u2014 extracting generic named entities, for example \u2014 maintaining high accuracy is a challenge. This is an ongoing area of exploration that we will report in the future.\nMachine learning is among major techniques for identifying candidate chunks. Besides HMM, we have also explored other classifiers such as the Naive Bayes classifier and neural networks-based classifier. An ongoing project includes a systematic study of different classifiers and their combinations (including Conditional Random Field and Support Vector Machine[27]) for online machine learning\u2013based data extraction.\nTo make it more flexible when using standard medical terminologies for customizing controlled vocabularies, a project is ongoing towards developing a tool that can easily search and import concepts from standard vocabularies such as ICDE-9, ICD-10, and SNOMED, from a local file or through NCBO BioPortal.\n\nConclusions \nAlthough there are natural language processing tools available for extracting information from clinical reports, the majority lack the capability to support interactive feedback from human users. An interactive, online approach allows the user to coach the system using knowledge specific to the given set of reports, which may include local reporting conventions and structures. Moreover, no advanced linguistics knowledge or programming skills are required of the users; the system maintains the ordinary workflow of manual annotation systems. We perform a systematic study on the effectiveness of the online learning\u2013based method combined with controlled vocabularies for data extraction from reports with various structural patterns, and we conclude that our method is highly effective. The framework is generic and the applicability is demonstrated with diverse report types. The software will be made freely available online.[28]\n\nAcknowledgments \nThe study was funded by Center for Disease Control and Prevention, contract #200-2014-M-59415.\n\nConflicts of interest \nNone declared.\n\nMultimedia Appendix 1 \nTest cases (PDF file; 32KB)\n\nReferences \n\n\n\u2191 Zheng, S. (07 September 2014). \"IDEAL-X: Information and Data Extraction using Adaptive Learning\". YouTube. Google, Inc. https:\/\/www.youtube.com\/watch?v=Q-DrWi31nv0 . Retrieved 20 April 2017 .   \n\n\u2191 2.0 2.1 Shalev-Shwartz, S. (July 2007). \"Online Learning: Theory, Algorithms, and Applications\" (PDF). University of Chicago. http:\/\/ttic.uchicago.edu\/~shai\/papers\/ShalevThesis07.pdf . Retrieved 03 May 2017 .   \n\n\u2191 3.0 3.1 Shalev-Shwartz, S. (2012). \"Online Learning and Online Convex Optimization\". Foundations and Trends in Machine Learning 4 (2): 107\u2013194. doi:10.1561\/2200000018.   \n\n\u2191 Smale, S.; Yao, Y. (2006). \"Online Learning Algorithms\". Foundations of Computational Mathematics 6 (2): 145\u2013170. doi:10.1007\/s10208-004-0160-z.   \n\n\u2191 Crowley, R.S.; Castine, M.; Mitchell, K. et al. (2010). \"caTIES: a grid based system for coding and retrieval of surgical pathology reports and tissue specimens in support of translational research\". JAMIA 17 (3): 253-64. doi:10.1136\/jamia.2009.002295. PMC PMC2995710. PMID 20442142. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995710 .   \n\n\u2191 Xu, H.; Stenner, S.P.; Doan, S. et al. (2010). \"MedEx: A medication information extraction system for clinical narratives\". JAMIA 17 (1): 19\u201324. doi:10.1197\/jamia.M3378. PMC PMC2995636. PMID 20064797. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995636 .   \n\n\u2191 Friedman, C.; Shagina, L.; Lussier, Y.; Hripcsak, G. (2004). \"Automated encoding of clinical documents based on natural language processing\". JAMIA 11 (5): 392\u2013402. doi:10.1197\/jamia.M1552. PMC PMC516246. PMID 15187068. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC516246 .   \n\n\u2191 Savova, G.K.; Masanz, J.J.; Ogren, P.V. et al. (2010). \"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): Architecture, component evaluation and applications\". JAMIA 17 (5): 507\u201313. doi:10.1136\/jamia.2009.001560. PMC PMC2995668. PMID 20819853. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995668 .   \n\n\u2191 Aronson, A.R. (2001). \"Effective mapping of biomedical text to the UMLS Metathesaurus: The MetaMap program\". Proceedings AMIA Symposium 2001: 17\u201321. PMC PMC2243666. PMID 11825149. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2243666 .   \n\n\u2191 Zeng, Q.T.; Goryachev, S.; Weiss, S. et al. (2006). \"Extracting principal diagnosis, co-morbidity and smoking status for asthma research: Evaluation of a natural language processing system\". BMC Medical Informatics and Decision Making 6: 30. doi:10.1186\/1472-6947-6-30. PMC PMC1553439. PMID 16872495. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1553439 .   \n\n\u2191 Hoi, S.C.H.; Wang, J.; Zhao, P. (2014). \"LIBOL: A Library for Online Learning Algorithms\". Journal of Machine Learning Research 15 (Feb): 495\u2013499. http:\/\/www.jmlr.org\/papers\/v15\/hoi14a.html .   \n\n\u2191 Cardie, C.; Pierce, D. (06 July 1998). \"Proposal for an Interactive Environment for Information Extraction\". Cornell University Computer Science Technical Report TR98-1702. Cornell University.   \n\n\u2191 Ciravegna, F.; Wilks, Y. (2003). \"Designing Adaptive Information Extraction for the Semantic Web in Amilcare\". In Handschuh, S.; Staab, S.. Annotation for the Semantic Web. 96. IOS Press. pp. 112\u2013127. ISBN 9781586033453.   \n\n\u2191 14.0 14.1 Ciravegna, F. (2001). \"Adaptive information extraction from text by rule induction and generalisation\". Proceedings of the 17th International Joint Conference on Artificial Intelligence 2: 1251\u20131256. ISBN 9781558608122.   \n\n\u2191 Gobbel, G.T.; Garvin, J.; Reeves, R. et al. (2014). \"Assisted annotation of medical free text using RapTAT\". JAMIA 21 (5): 833-41. doi:10.1136\/amiajnl-2013-002255. PMC PMC4147611. PMID 24431336. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4147611 .   \n\n\u2191 Settles, B. (January 2009). \"Active Learning Literature Survey\" (PDF). University of Wisconsin. https:\/\/research.cs.wisc.edu\/techreports\/2009\/TR1648.pdf . Retrieved 03 May 2017 .   \n\n\u2191 Olsson, F. (17 April 2009). \"A literature survey of active machine learning in the context of natural language processing\" (PDF). Swedish Institute of Computer Science. http:\/\/soda.swedish-ict.se\/3600\/1\/SICS-T--2009-06--SE.pdf . Retrieved 03 May 2017 .   \n\n\u2191 Settles, B. (2011). \"Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances\". Proceedings of the Conference on Empirical Methods in Natural Language Processing 2011: 1467\u20131478. ISBN 9781937284114.   \n\n\u2191 Chen, Y.; Cao, H.; Mei, Q. et al. (2013). \"Applying active learning to supervised word sense disambiguation in MEDLINE\". JAMIA 2013: 1001\u20131006. doi:10.1136\/amiajnl-2012-001244. PMC PMC3756255. PMID 23364851. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3756255 .   \n\n\u2191 Chen, Y.; Carroll, R.J.; Hinz, E.R. et al. (2013). \"Applying active learning to high-throughput phenotyping algorithms for electronic health records data\". JAMIA 20 (e2): e253-9. doi:10.1136\/amiajnl-2013-001945. PMC PMC3861916. PMID 23851443. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3861916 .   \n\n\u2191 Eapen, D.J.; Manocha, P.; Patel, R.S. et al. (2013). \"Aggregate risk score based on markers of inflammation, cell stress, and coagulation is an independent predictor of adverse cardiovascular outcomes\". Journal of the American College of Cardiology 62 (4): 329-37. doi:10.1016\/j.jacc.2013.03.072. PMC PMC4066955. PMID 23665099. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4066955 .   \n\n\u2191 \"OpenNLP\". Apache Software Foundation. http:\/\/opennlp.apache.org\/ . Retrieved 20 April 2017 .   \n\n\u2191 23.0 23.1 23.2 Manning, C.D.; Raghavan, P.; Sch\u00fctze, H. (2008). Introduction to Information Retrieval. Cambridge University Press. pp. 506. ISBN 9780521865715.   \n\n\u2191 24.0 24.1 Elliott, R.J.; Aggoun, L.; Moore, J.B. (2008). Hidden Markov Models: Estimation and Control. Springer. pp. 382. ISBN 9780387943640.   \n\n\u2191 25.0 25.1 F\u00fcrnkranz, J. (1999). \"Separate-and-Conquer Rule Learning\". Artificial Intelligence Review 13 (1): 3\u201354. doi:10.1023\/A:1006524209794.   \n\n\u2191 Huang, Y.; Lowe, H.J. (2007). \"A novel hybrid approach to automated negation detection in clinical radiology reports\". JAMIA 14 (3): 304\u201311. doi:10.1197\/jamia.M2284. PMC PMC2244882. PMID 17329723. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2244882 .   \n\n\u2191 Jiang, M.; Chen, Y.; Liu, M. et al. (2011). \"A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries\". JAMIA 18 (5): 601-6. doi:10.1136\/amiajnl-2011-000163. PMC PMC3168315. PMID 21508414. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3168315 .   \n\n\u2191 \"IDEAL-X\". Emory University. http:\/\/www.idealx.net\/home.html . Retrieved 04 May 2017 .   \n\n\nAbbreviations \nEMR: electronic medical record\nHMM: Hidden Markov Model\nNLP: natural language processing\nPOS: part of speech\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID was missing and was added to make the reference more useful. Grammar and vocabulary were cleaned up to make the article easier to read. In the \"Answer prediction\" subsection, the original referred to Figure 5, when it most likely should have referred to Figure 4; this was changed for this version.\nPer the distribution agreement, the following copyright information is also being added: \n\u00a9Shuai Zheng, James J Lu, Nima Ghasemzadeh, Salim S Hayek, Arshed A Quyyumi, Fusheng Wang. Originally published in JMIR Medical Informatics (http:\/\/medinform.jmir.org), 09.05.2017.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\">https:\/\/www.limswiki.org\/index.php\/Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 23 May 2017, at 17:01.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,106 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","a67695148af73db7d41c4171d7c86cbc_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Effective information extraction framework for heterogeneous clinical reports using online machine learning and controlled vocabularies<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: Extracting structured data from narrated medical reports is challenged by the complexity of heterogeneous structures and vocabularies and often requires significant manual effort. Traditional machine-based approaches lack the capability to take user feedback for improving the extraction algorithm in real time.\n<\/p><p><b>Objective<\/b>: Our goal was to provide a generic <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> extraction framework that can support diverse clinical reports and enables a dynamic interaction between a human and a machine that produces highly accurate results.\n<\/p><p><b>Methods<\/b>: A clinical information extraction system IDEAL-X has been built on top of online machine learning. It processes one document at a time, and user interactions are recorded as feedback to update the learning model in real time. The updated model is used to predict values for extraction in subsequent documents. Once prediction accuracy reaches a user-acceptable threshold, the remaining documents may be batch processed. A customizable controlled vocabulary may be used to support extraction.\n<\/p><p><b>Results<\/b>: Three datasets were used for experiments based on report styles: 100 cardiac catheterization procedure reports, 100 coronary angiographic reports, and 100 integrated reports \u2014 each combines history and physical report, discharge summary, outpatient clinic notes, outpatient clinic letter, and inpatient discharge medication report. Data extraction was performed by three methods: online machine learning, controlled vocabularies, and a combination of these. The system delivers results with F1 scores greater than 95%.\n<\/p><p><b>Conclusions<\/b>: IDEAL-X adopts a unique online machine learning\u2013based approach combined with controlled vocabularies to support data extraction for clinical reports. The system can quickly learn and improve, thus it is highly adaptable.\n<\/p><p><b>Keywords<\/b>: information extraction, natural language processing, controlled vocabulary, electronic medical records \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>While immense efforts have been made to enable a structured data model for <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" target=\"_blank\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical records<\/a> (EMRs), a large amount of medical data remain in free-form narrative text, and useful data from individual patients are usually distributed across multiple reports of heterogeneous structures and vocabularies. This poses major challenges to traditional information extraction systems, as either costly training datasets or manually crafted rules have to be prepared. These approaches also lack the capability of taking user feedback to adapt and improve the extraction algorithm in real time.\n<\/p><p>Our goal is to provide a generic information extraction framework that adapts to diverse clinical reports, enables a dynamic interaction between a human and a machine, and produces highly accurate results with minimal human effort. We have developed a system, Information and Data Extraction using Adaptive Online Learning (IDEAL-X), to support adaptive information extraction from diverse clinical reports with heterogeneous structures and vocabularies. The system is built on top of online machine learning and customizable controlled vocabularies. A demo video can be found on YouTube.<sup id=\"rdp-ebb-cite_ref-ZhengIDEAL14_1-0\" class=\"reference\"><a href=\"#cite_note-ZhengIDEAL14-1\" rel=\"external_link\">[1]<\/a><\/sup>\n<\/p><p>IDEAL-X uses an online machine learning\u2013based approach<sup id=\"rdp-ebb-cite_ref-Shalev-ShwartzOnline07_2-0\" class=\"reference\"><a href=\"#cite_note-Shalev-ShwartzOnline07-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Shalev-ShwartzOnline12_3-0\" class=\"reference\"><a href=\"#cite_note-Shalev-ShwartzOnline12-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmaleOnline06_4-0\" class=\"reference\"><a href=\"#cite_note-SmaleOnline06-4\" rel=\"external_link\">[4]<\/a><\/sup> for information extraction. Traditional machine learning algorithms take a two-stage approach: batch training based on an annotated training dataset, and batch prediction for future datasets based on the model generated from stage one (Figure 1). In contrast, online machine learning algorithms<sup id=\"rdp-ebb-cite_ref-Shalev-ShwartzOnline07_2-1\" class=\"reference\"><a href=\"#cite_note-Shalev-ShwartzOnline07-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Shalev-ShwartzOnline12_3-1\" class=\"reference\"><a href=\"#cite_note-Shalev-ShwartzOnline12-3\" rel=\"external_link\">[3]<\/a><\/sup> take an iterative approach (Figure 1). IDEAL-X learns one document at a time and predicts values to be extracted for the next one. Learning occurs from revisions made by the user, and the updated model is applied to prediction for subsequent documents. Once the model achieves a satisfactory accuracy, the remaining documents may be processed in batches. Online machine learning not only significantly reduces user effort for annotation but also provides the mechanism for collecting feedback from human-machine interaction to improve the system's model continuously.\n<\/p><p>Besides online machine learning, IDEAL-X allows for customizable controlled vocabularies to support data extraction from clinical reports, where a vocabulary enumerates the possible values that can be extracted for a given attribute. (The X in IDEAL-X represents the controlled vocabulary plug-in.) The use of online machine learning and controlled vocabularies is not mutually exclusive; they are complementary, which provides the user with a variety of modes for working with IDEAL-X.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Zheng_JMIRMedInfo2017_5-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"0ccf8fc01db3a276007426c6d69018ab\"><img alt=\"Fig1 Zheng JMIRMedInfo2017 5-2.png\" src=\"https:\/\/www.limswiki.org\/images\/7\/71\/Fig1_Zheng_JMIRMedInfo2017_5-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> Online machine learning versus batch learning. (a) Batch machine learning workflow; (b) Online machine learning workflow<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Related_work\">Related work<\/span><\/h3>\n<p>A number of research efforts have been made in different fields of medical information extraction. Successful systems include caTIES<sup id=\"rdp-ebb-cite_ref-Crowley_caTIES10_5-0\" class=\"reference\"><a href=\"#cite_note-Crowley_caTIES10-5\" rel=\"external_link\">[5]<\/a><\/sup>, MedEx\n<sup id=\"rdp-ebb-cite_ref-XuMedEx10_6-0\" class=\"reference\"><a href=\"#cite_note-XuMedEx10-6\" rel=\"external_link\">[6]<\/a><\/sup>, MedLEE<sup id=\"rdp-ebb-cite_ref-FriedmanAuto04_7-0\" class=\"reference\"><a href=\"#cite_note-FriedmanAuto04-7\" rel=\"external_link\">[7]<\/a><\/sup>, cTAKES<sup id=\"rdp-ebb-cite_ref-SavovaMayo10_8-0\" class=\"reference\"><a href=\"#cite_note-SavovaMayo10-8\" rel=\"external_link\">[8]<\/a><\/sup>, MetaMap<sup id=\"rdp-ebb-cite_ref-AronsonEffective01_9-0\" class=\"reference\"><a href=\"#cite_note-AronsonEffective01-9\" rel=\"external_link\">[9]<\/a><\/sup>, HITEx<sup id=\"rdp-ebb-cite_ref-ZengExtracting06_10-0\" class=\"reference\"><a href=\"#cite_note-ZengExtracting06-10\" rel=\"external_link\">[10]<\/a><\/sup>, and so on. These methods either take a rule-based approach, a traditional machine learning\u2013based approach, or a combination of both.\n<\/p><p>Different online learning algorithms have been studied and developed for classification tasks<sup id=\"rdp-ebb-cite_ref-HoiLIBOL14_11-0\" class=\"reference\"><a href=\"#cite_note-HoiLIBOL14-11\" rel=\"external_link\">[11]<\/a><\/sup>, but their direct application to information extraction has not been studied. Especially in the clinical environment, the effectiveness of these algorithms is yet to be examined. Several pioneering projects have used learning processes that involve user interaction and certain elements of IDEAL-X. I<sup>2<\/sup>E<sup>2<\/sup> is an early rule-based interactive information extraction system.<sup id=\"rdp-ebb-cite_ref-CardieProposal98_12-0\" class=\"reference\"><a href=\"#cite_note-CardieProposal98-12\" rel=\"external_link\">[12]<\/a><\/sup> It is limited by its restriction to a predefined feature set. Amilcare<sup id=\"rdp-ebb-cite_ref-CiravegnaDesigning03_13-0\" class=\"reference\"><a href=\"#cite_note-CiravegnaDesigning03-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CiravegnaAdaptive01_14-0\" class=\"reference\"><a href=\"#cite_note-CiravegnaAdaptive01-14\" rel=\"external_link\">[14]<\/a><\/sup> is adaptable to different domains. Each domain requires an initial training that can be retrained on the basis of the user\u2019s revision. Its algorithm (LP)<sup>2<\/sup> is able to generalize and induce symbolic rules. RapTAT<sup id=\"rdp-ebb-cite_ref-GobbelAssisted14_15-0\" class=\"reference\"><a href=\"#cite_note-GobbelAssisted14-15\" rel=\"external_link\">[15]<\/a><\/sup> is most similar to IDEAL-X in its goals. It preannotates text interactively to accelerate the annotation process. It uses a multinominal na\u00efve Baysian algorithm for classification but does not appear to use contextual information beyond previously found values in its search process. This may limit its ability to extract certain value types.\n<\/p><p>Different from online machine learning but related is active learning<sup id=\"rdp-ebb-cite_ref-SettlesActive09_16-0\" class=\"reference\"><a href=\"#cite_note-SettlesActive09-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OlssonALit09_17-0\" class=\"reference\"><a href=\"#cite_note-OlssonALit09-17\" rel=\"external_link\">[17]<\/a><\/sup>; it assumes the ability to retrieve labels for the most informative data points while involving the users in the annotation process. DUALIST<sup id=\"rdp-ebb-cite_ref-SettlesClosing11_18-0\" class=\"reference\"><a href=\"#cite_note-SettlesClosing11-18\" rel=\"external_link\">[18]<\/a><\/sup> allows users to select system-populated rules for feature annotation to support information extraction. Other example applications in <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">health care informatics<\/a> include word sense disambiguation<sup id=\"rdp-ebb-cite_ref-ChenApplyingMEDLINE13_19-0\" class=\"reference\"><a href=\"#cite_note-ChenApplyingMEDLINE13-19\" rel=\"external_link\">[19]<\/a><\/sup> and phenotyping.<sup id=\"rdp-ebb-cite_ref-ChenApplyingEHR13_20-0\" class=\"reference\"><a href=\"#cite_note-ChenApplyingEHR13-20\" rel=\"external_link\">[20]<\/a><\/sup> Active learning usually requires comprehending the entire corpus in order to pick the most useful data point. However, in a clinical environment, data arrive in a streaming fashion over time, which limits our ability to choose data points. Hence, an online learning approach is more suitable.\n<\/p><p>IDEAL-X adopts the Hidden Markov Model for its compatibility with online learning, and for its efficiency and scalability. We will also describe a broader set of contextual information used by the learning algorithm to facilitate extraction of values of all types.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Heterogeneous_clinical_reports\">Heterogeneous clinical reports<\/span><\/h3>\n<p>A patient\u2019s electronic medical record could come with a variety of medical reports. Data in these reports provide critical information that can be used to improve clinical diagnosis and support biomedical research. For example, the Emory University Cardiovascular Biobank<sup id=\"rdp-ebb-cite_ref-EapenAggregate13_21-0\" class=\"reference\"><a href=\"#cite_note-EapenAggregate13-21\" rel=\"external_link\">[21]<\/a><\/sup> collects records of patients with potential or confirmed coronary artery diseases undergoing cardiac catheterization, and it aims to combine extracted data elements from multiple reports to identity patients for research. Report types include history and physical report, discharge summary, outpatient clinic note, outpatient clinic letter, coronary angiogram report, cardiac catheterization procedure report, echocardiogram report, inpatient report, and discharge medication lists.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Zheng_JMIRMedInfo2017_5-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"f21577ba6c554fb91bc20ad0491fff5b\"><img alt=\"Fig2 Zheng JMIRMedInfo2017 5-2.png\" src=\"https:\/\/www.limswiki.org\/images\/a\/af\/Fig2_Zheng_JMIRMedInfo2017_5-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> Example snippets of different report forms. (a) Semi-structured report; (b) Template based narration; and (c) Complex narration<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>We classify clinical reports into three forms: semi-structured data, template-based narration, and complex narration. Semi-structured data represent data elements in the form of attribute and value pairs (Figure 2). Reports in this form have simple structures, making data extraction relatively straightforward. Template-based narration is a very common report form. The narrative style, including sentence patterns and vocabularies, follows consistent templates and expressions (Figure 2). Extracting information from this type of text (e.g., \"right posterior descending artery\") require major linguistics expertise, to either formulate extraction rules or to annotate training data. Complex narration is essentially free-form text. It can be irregular, personal, and idiomatic (Figure 2). Most medical reporting systems still allow for (and thus encourage) such a style. It is the most difficult form to interpret and process by natural language processing (NLP) algorithms. Nevertheless, certain types of information such as diseases and medications have finite vocabulary that could be used to support data extraction.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Overview\">Overview<\/span><\/h3>\n<p>The interface and workflow conform to traditional annotation systems: a user browses an input document from the input document collection and fills out an output form. On loading each document, the system attempts to fill the output form automatically with its data extraction engine. Then, a user can review and revise incorrect answers. The system then updates its data extraction model automatically based on the user\u2019s feedback. Optionally, the user may provide a customized controlled vocabulary to further support data extraction and answer normalization. Pretraining with manually annotated data is not required, as the prediction model behind the data extraction engine can be established incrementally through online learning, customizing controlled vocabularies, or a combination of the two.\n<\/p><p>The system can operate in two modes: (1) interactive: through online learning, the system predicts values to be extracted for each report, and the user verifies or corrects the predicted values; and (2) batch: batch predicting for all unprocessed documents once the accrued accuracy is sufficient for users. Whereas interactive mode uses online machine learning to build the learning model incrementally, batch mode runs the same as the prediction phase of batch machine learning.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"System_interface_and_user_operations\">System interface and user operations<\/span><\/h4>\n<p><b>System interface<\/b>\n<\/p><p>IDEAL-X provides a GUI with two main panels, a menu, and navigation buttons (Figure 3). The left panel is for browsing an input report, and the right panel is the output table with predicted values of each data element in the report. The menu provides options for defining the data elements to be extracted, specifying input reports, among others.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Zheng_JMIRMedInfo2017_5-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"32cdf77332f39ea3844762e609df56d3\"><img alt=\"Fig3 Zheng JMIRMedInfo2017 5-2.png\" src=\"https:\/\/www.limswiki.org\/images\/a\/a6\/Fig3_Zheng_JMIRMedInfo2017_5-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3<\/b> An example screenshot of IDEAL-X\u2019s interface<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><b>Definition of data elements for extraction<\/b>\n<\/p><p>The system provides a wizard for constructing the metadata of the output form. The user builds the form by specifying a list of data elements and their constraints. An example is the data element \"Heart Rate,\" which is constrained to be a numerical value between 0 and 200. Other constraints include sections of the report that may contain the values. However, except for the names of the data elements, specifying constraints are optional, as these can be learned by the system.\n\nData extraction workflow\n<\/p><p>The user will first select a collection of input reports to be extracted from a local folder. By default, the system runs in an interactive mode, and one report will be loaded at a time on the left display panel. The user can make manual annotations by highlighting the correct value in the report text. Clicking the corresponding data field in the table assigns the value to the data element. If the system has prefilled the field of a data element with a predicted value, the user can provide feedback by fixing incorrect values. As the user navigates to the next document, the system compares the prefilled and the final values for the most recently processed document. Values that are unchanged or filled in by users are taken as positive instances, and values that have been revised are taken as negative instances. Both instances are incorporated into the online learning algorithm to be used by the data extraction for subsequent documents. By iterating through this process, the amount of information that the system is able to correctly prefill grows over time. Note that manual revision in this context is different from traditional human labeling. It is only necessary if there is a wrong prediction, thus user effort can be significantly saved. Once the decision model reaches an acceptable level of accuracy, the user has the option to switch to batch mode to complete extraction for the remaining documents. If a patient has multiple reports, the text input panel displays each report with a separate tab. Data extracted from all the reports are aggregated in the output.\n\nCustomization of controlled vocabularies\n<\/p><p>IDEAL-X also provides an interface for the user to customize a controlled vocabulary that can be used by the system for data extraction. The controlled vocabulary contains both terminology and structural properties. The terminology includes lists of values and their normalization mappings. For example, disease terminology includes \"Diabetes Mellitus\" with variations \"DM\" and \"Diabetes.\" It also defines inductions. For example, taking \"Insulin\" or \"Metformin\" indicates having diabetes mellitus. Structural properties provide positive and negative contextual information for giving terms. For example, to extract medications taken by patients, the \"Allergies\" section is a negative context and medicine names in the section will be skipped. Structural properties may also contain disambiguation terms that may further improve the precision of extraction. A simple example is that \"intolerant\" is a negative indicator for identifying \"statin\" as \"statin intolerant\" refers to a different concept. Controlled vocabularies can be a powerful tool to support data extraction: it can be used to locate sentences and chunks of possible values, and to perform normalization for extracted values, discussed in the next section.\n<\/p><p><b>The data extraction engine<\/b>\n<\/p><p>While the user interacts with IDEAL-X interface, the data extraction engine works transparently in the background. The engine has three major components: answer prediction, learning, and the learning model that the online learning process continuously updates (Figure 4). The system combines statistical and machine learning\u2013based approaches with controlled vocabularies for effective data extraction.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Document_preprocessing\">Document preprocessing<\/span><\/h4>\n<p>When a report is loaded, the text is first parsed into an in-memory hierarchical tree consisting of four layers: section, paragraph, sentence, and token. Apache OpenNLP<sup id=\"rdp-ebb-cite_ref-OpenNLP_22-0\" class=\"reference\"><a href=\"#cite_note-OpenNLP-22\" rel=\"external_link\">[22]<\/a><\/sup> is used to support the parsing with its Sentence Detector, Tokenizer, and Part-of-Speech Tagger. A reverse index of tokens is created to support an efficient keyword-based search. The index is used to find locations (e.g., sections, paragraphs, sentences, and phrases) of a token, as well as its properties such as part of speech and data type. For example, given the token \"DM,\" the system can quickly identify the section (e.g., \"History\") and the containing sentences. Such token search is frequently performed in answer prediction, and the in-memory index structures enable high efficiency for such operations.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Answer_prediction\">Answer prediction<\/span><\/h4>\n<p>Predicting the value of each data element involves the following steps: (1) Identifying target sentences that are likely to contain the answer; (2) Identifying candidate chunks in the sentences; (3) Filtering the chunks to generate candidate values; (4) Ranking candidate values to generate (raw) values; (5) Normalizing values; and (6) Aggregating values from multiple reports. The workflow is shown in Figure 4.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Zheng_JMIRMedInfo2017_5-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"2493c4434526a5a571ff1df6a20f54f3\"><img alt=\"Fig4 Zheng JMIRMedInfo2017 5-2.png\" src=\"https:\/\/www.limswiki.org\/images\/5\/57\/Fig4_Zheng_JMIRMedInfo2017_5-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 4<\/b> Overview of System Workflow<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><b>Identifying target sentences<\/b>\n<\/p><p>Through online learning, the system accrues keywords from past answers (answer keywords) along with co-occurring words in the corresponding sentences (contextual words). For example, given the answer keywords \"diabetes\" and \"hypertension\" in the sentence \"The patient reports history of diabetes and hypertension,\" contextual words are \"patient,\" \"report,\" and \"history.\" Such answer keywords and contextual words combined with customized vocabularies can be utilized to identify sentences that are likely to contain answers with the following methods:\n<\/p><p>First, there's similarity-based search using the vector space model.<sup id=\"rdp-ebb-cite_ref-ManningIntro08_23-0\" class=\"reference\"><a href=\"#cite_note-ManningIntro08-23\" rel=\"external_link\">[23]<\/a><\/sup> Given a collection of contextual words and their frequencies, the system computes the similarity against sentences in the document.<sup id=\"rdp-ebb-cite_ref-ManningIntro08_23-1\" class=\"reference\"><a href=\"#cite_note-ManningIntro08-23\" rel=\"external_link\">[23]<\/a><\/sup> Sentences with high similarities are selected. For example, most sentences about \"disease\" contain \"diagnosis\" and \"history.\" The past contextual keywords and their frequency weights are represented and maintained through a learning model discussed later in section on Learning.\n<\/p><p>Second, we have the answer keyword matching search. The answer keywords, combined with relevant user customized vocabularies, are also used to identify target sentences with keyword matching. For example, to extract diseases, if a sentence contains the disease term \"myocardial infarction\" defined in the vocabulary, the sentence is selected as a target. In both approaches, sections to be searched or skipped are also considered to narrow the scope of searching.\n<\/p><p><b>Identifying candidate chunks<\/b>\n<\/p><p>After target sentences are selected, the system identifies potential phrases in the sentences using two methods: Hidden Markov model (HMM)<sup id=\"rdp-ebb-cite_ref-ElliottHidden08_24-0\" class=\"reference\"><a href=\"#cite_note-ElliottHidden08-24\" rel=\"external_link\">[24]<\/a><\/sup> and keyword-based search. The HMM represents target words and contextual words in a sentence with different states, and it marks values to be extracted based on probability distributions learned from previously collected values and their sentences. The keyword-based search finds candidate chunks using keywords collected from past answers and the controlled vocabulary.\n\nFiltering chunks\n<\/p><p>To filter candidate chunks, the system uses rule induction<sup id=\"rdp-ebb-cite_ref-CiravegnaAdaptive01_14-1\" class=\"reference\"><a href=\"#cite_note-CiravegnaAdaptive01-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-F.C3.BCrnkranzSeparate99_25-0\" class=\"reference\"><a href=\"#cite_note-F.C3.BCrnkranzSeparate99-25\" rel=\"external_link\">[25]<\/a><\/sup> to generate \"If-Then\" rules based on historical statistics. The following filtering criteria are used: (1) Part of speech (POS): This filters a phrase by its POS tag in the sentence. Simple example phrases are noun and verb phrases. (2) String pattern: This looks for chunks that match special string patterns. For example, the first characters of all tokens are capitalized. (3) Value domain: This eliminates numerical or enumerated values that fall outside a specified range of values. (4) Negation: Based on predefined built-in rules, this removes phrases governed by words that reverse the meaning of the answer.<sup id=\"rdp-ebb-cite_ref-HuangANovel07_26-0\" class=\"reference\"><a href=\"#cite_note-HuangANovel07-26\" rel=\"external_link\">[26]<\/a><\/sup> For example, if a candidate chunk \"cancer\" is extracted from a sentence \"the patient has no history of cancer,\" \"cancer\" would not be included. (5) Certainty: Similar to negation filter, this detects and filters uncertain events or situations such as future plans, based on predefined rules. For example, a candidate chunk \"radiation therapy\" for treatment from a sentence \"the patient is planned to take radiation therapy\" should not be included. Whereas negation and certainty filtering is based on predefined rules, other filtering relies on real-time data statistics for filtering criteria.\n\nRanking candidate values\n<\/p><p>The system combines the scores of the selected sentences and chunks for ranking of candidate values. For a single-valued data element (e.g., heart beat), the candidate value with the highest confidence score is selected. For a multi-valued data element (e.g., medication), values with confidence scores above a threshold are selected. Based on this, each candidate value is either accepted or rejected.\n<\/p><p><b>Normalizing values<\/b>\n<\/p><p>This step normalizes extracted values through transformation, generalization, and induction rules given by the controlled vocabulary (Figure 4). For example, \"DM\" is transformed into \"Diabetes Mellitus.\" \"Pindolol\" is generalized to its hypernym \"beta blocker.\" The appearance of the medication term \"Metformin\" (a drug for treating type 2 diabetes) in the text can infer the disease \"Diabetes Mellitus.\"\n<\/p><p><b>Aggregating results<\/b>\n<\/p><p>Data extracted from multiple reports of a patient will be aggregated into a single table. The aggregation process may normalize values and remove duplicates. For example, \"lisinopril\" and \"captopril\" are extracted from discharge summary and inpatient report, respectively, and they can be normalized as \"ACE inhibitor.\" If the same data element is extracted from multiple reports, deduplication is performed. The final output is in simple structural table form that can be exported conveniently to other applications such as Excel (Microsoft) or a database.\n<\/p><p>Note that controlled vocabularies can play important roles in the answer prediction process. They are used for identifying target sentences through keyword searching, identifying candidate chunks through keyword matching, and supporting normalization for extracted values.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Learning\">Learning<\/span><\/h4>\n<p>DEAL-X takes an online learning\u2013based approach to incrementally build statistical models and make predictions (Figure 5). The three models used in IDEAL-X are all statistical based and can be continuously updated after each iteration.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Zheng_JMIRMedInfo2017_5-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"40b37e55cef6ad9ab2cb678430b212b3\"><img alt=\"Fig5 Zheng JMIRMedInfo2017 5-2.png\" src=\"https:\/\/www.limswiki.org\/images\/1\/11\/Fig5_Zheng_JMIRMedInfo2017_5-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 5<\/b> Precision and recall changes over processed records<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>System-predicted values automatically populate the output table, and the user advances to the next report with or without revision to these values. In both cases, the internal learning and prediction models of IDEAL-X are updated. For each instance, IDEAL-X collects and analyzes the following features: (1) Position: location of the answer in the text hierarchy; (2) Landmark: co-occurring contextual keywords in a sentence; (3) POS: parts of speech tag; (4) Value: the tokens of the answer; and (5) String patterns: literal features such as capitalization and initial and special punctuation. These features are then used to update the three models.\n<\/p><p>In IDEAL-X, each data element such as attribute \"disease\" or \"medicine\" has its own statistical model, and each new instance of a data element will update the corresponding model. There are three models to be updated: (1) Updating Space Vector Model: This model uses \"Landmark\" features of positive instances. The system updates frequencies of co-occurring contextual words, used as weights of the space vector.<sup id=\"rdp-ebb-cite_ref-ManningIntro08_23-2\" class=\"reference\"><a href=\"#cite_note-ManningIntro08-23\" rel=\"external_link\">[23]<\/a><\/sup> (2) Updating HMM: HMM lists all words in a sentence as a sequence, in which an extracted value is marked as target value state and other words are recognized as irrelevant contextual states. Based on this sequence, the state transition probabilities and emission probabilities are recalculated.<sup id=\"rdp-ebb-cite_ref-ElliottHidden08_24-1\" class=\"reference\"><a href=\"#cite_note-ElliottHidden08-24\" rel=\"external_link\">[24]<\/a><\/sup> (3) Updating rule induction model: Filtering rules are induced based on the coverage percentage.<sup id=\"rdp-ebb-cite_ref-F.C3.BCrnkranzSeparate99_25-1\" class=\"reference\"><a href=\"#cite_note-F.C3.BCrnkranzSeparate99-25\" rel=\"external_link\">[25]<\/a><\/sup> Features such as POS, value domain, and string patterns of both positive and negative instances are analyzed and their respective coverage percentages are modified. Once the coverage of a rule reaches a predefined threshold, the rule is triggered for filtering.\n<\/p><p>In an interactive mode, the above four steps repeat for each report, where the learning models are continuously updated and improved.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Experimental_setup\">Experimental setup<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Datasets\">Datasets<\/span><\/h4>\n<p>We used three datasets from 100 patients that were randomly sampled from a collection of about 5000 patients in the Emory Biobank database. Dataset 1 is a set of semi-structured reports and contains 100 cardiac catheterization procedure reports. Dataset 2 is a set of template-based narration and contains 100 coronary angiographic reports. Dataset 3 is a set of complex narration and contains 315 reports, including history and physical report, discharge summary, outpatient clinic notes, outpatient clinic letter, and inpatient discharge medication report.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Ground_truth\">Ground truth<\/span><\/h4>\n<p>The test datasets are independently hand-annotated by domain expert annotators, including physicians, physician trainees, and students trained by the Emory Clinical Cardiovascular Research Institute for Biobank data reporting. Each record is annotated by two different annotators. The interrater agreement scores (kappa) of these three datasets are .991, .986, and .835, respectively. An arbitrator \u2014 an independent cardiovascular disease researcher \u2014 reconciles incompatible outputs of the system and the manual annotations to produce the final ground truth.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Evaluation_metrics\">Evaluation metrics<\/span><\/h4>\n<p>For validation, precision, recall, and F1, scores are used to estimate the effectiveness of extraction by comparing the system predicted results (before human revision) and the ground truth.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Experiment_settings\">Experiment settings<\/span><\/h4>\n<p>We aimed to evaluate the effectiveness of the system with respect to using online learning and controlled vocabularies and to understand their applicability to different report forms. By analyzing the report styles and vocabularies, we discovered that online learning will be more suitable for semi-structured or template-based narration reports, and controlled vocabulary-guided data extraction would be more effective on complex narration with a finite vocabulary. Thus, we designed three experiments: (1) Online learning\u2013based data extraction, where controlled vocabularies are not provided, based on Dataset 1 (semi-structured) and Dataset 2 (template-based narration); (2) Controlled vocabularies-based data extraction, where online learning is not used, based on Dataset 3 (complex narration); and (3) Controlled vocabularies-guided data extraction combined with online learning, based on Dataset 3.\n<\/p><p><b>Performance evaluation<\/b>\n<\/p><p><i>Experiment 1: Online machine learning\u2013based data extraction<\/i>: This experiment was based on Datasets 1 and 2. The system starts in an interactive mode with an empty decision model without prior training. The defined data elements are summarized in Multimedia Appendix 1. The user processes one report at a time, and each system-predicted value (including empty values for the first few reports) before user revision was recorded for calculating precision and recall.\n<\/p><p>Results are summarized in Table 1 for the two datasets, respectively. Both test cases achieved high precision as semi-structured and template-based text is most easy to handle. To study the learning rate of online learning, we divided records into 10 groups and plotted precision and recall of every 10 percent of the records in Datasets 1 and 2. We observed that in both tests, the system maintained high precision during the learning process. Although some variability exists due to new data pattern, the recall of both cases also improved steadily. Not surprisingly, the rate of learning for Dataset 1 is much faster given its semi-structure.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 1.<\/b> Results of data extraction from semistructured reports (Dataset 1) and template-based narration (Dataset 2)\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Dataset\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of data elements\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of values\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Precision (%)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Recall (%)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">F1 (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1272\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">728\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">93.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">95.2\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><i>Experiment 2: Controlled vocabularies-guided data extraction<\/i>: In this experiment, online learning was disabled and data extraction was performed in batches using controlled vocabulary. Diseases and medications were extracted from Dataset 3 (values to be extracted are shown in Multimedia Appendix 1). Customized controlled vocabularies, including terminology and structural properties, had been created independently by physicians via references to domain knowledge resources and analyzing another development report dataset of 100 patients, disjointed from Dataset 3. Note that comparisons in this and the following experiments were at a clinical finding level between system-integrated results and manual-annotation-integrated results.\n<\/p><p>The results in Table 2 show that controlled vocabularies are highly effective for data extraction over complex narratives. Domain-specific data such as cardiology-related diseases and medications have limited numbers of possible values (or domain values), and a carefully customized controlled vocabulary can achieve high extraction accuracy.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 2.<\/b> Results of controlled vocabularies-guided data extraction from complex narration (Dataset 3)\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Type of data elements\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of data elements\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of round truth values\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Precision (%)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Recall (%)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">F1 (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diseases\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">418\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">94.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96.7\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medications\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">437\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">All\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">855\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97.9\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><i>Experiment 3: Controlled vocabularies-guided data extraction combined with online machine learning<\/i>: In this experiment, we performed two tests to examine how efficient and effective the system learns when only terminology is available and structural properties need to be obtained from online learning. Test 1 was to generate the baseline for comparison, and Test 2 was to demonstrate the effectiveness of combining online machine learning and controlled vocabularies. Dataset 3 was used to extract all diseases and medications.\n<\/p><p>For Test 1, terminology was used and online machine learning was disabled, so the test was guided by controlled vocabulary without any structural properties. We note that comprehensive terminology contributes directly to high recall rate, which means that the system seldom misses values to be extracted. However, if structural properties are not included, compared with the result in Experiment 2, the precision is much lower. This highlights the value of positive and negative contexts in an extraction task.\n<\/p><p>For Test 2, both terminology and online machine learning were used. Online machine learning supports learning structural properties. To show how quickly the system learns, only the 38 reports associated with the first 10 patients were processed with interactive online learning. All remaining reports were processed in batch. Results in Table 3 show an overall precision of 94.9%, which demonstrates that online learning could quickly learn structural properties.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 3.<\/b> Results of controlled vocabularies-guided data extraction combined with online learning\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Test\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Controlled vocabulary\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Online learning\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Precision (%)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Recall (%)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">F1 (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Terminology only\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\/A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">89.2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Terminology only\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Applied to first 10 patients\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">94.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97.1\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Typical errors in these two tests were associated with terminology and contextual information used in complex narrative scenarios. On one hand, the completeness of the terminology list, including terms and their synonyms, influences the recall rate directly. On the other hand, although coverage of terminologies could be maximized by a carefully engineered vocabulary, unwanted extractions arising from searches in the wrong section, undetected negations, and ambiguous use of terms can still lower the overall precision.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Principal_findings\">Principal findings<\/span><\/h3>\n<p>IDEAL-X provides a generic data extraction framework that takes advantage of both online learning and controlled vocabularies. The two approaches complement each other and can also be combined. An online learning\u2013based approach is highly effective for reports with underlying structural patterns such as semi-structured or template-based narration style-based reports. Experiments with complex narrative reports indicate that the use of controlled vocabularies is highly effective for supporting extraction constrained by a finite data domain. In addition, structural properties such as section-data associations can play an important role in improving the accuracy of extraction. However, in cases where controlled vocabularies are unavailable \u2014 extracting generic named entities, for example \u2014 maintaining high accuracy is a challenge. This is an ongoing area of exploration that we will report in the future.\n<\/p><p>Machine learning is among major techniques for identifying candidate chunks. Besides HMM, we have also explored other classifiers such as the Naive Bayes classifier and neural networks-based classifier. An ongoing project includes a systematic study of different classifiers and their combinations (including Conditional Random Field and Support Vector Machine<sup id=\"rdp-ebb-cite_ref-JiangAStudy11_27-0\" class=\"reference\"><a href=\"#cite_note-JiangAStudy11-27\" rel=\"external_link\">[27]<\/a><\/sup>) for online machine learning\u2013based data extraction.\n<\/p><p>To make it more flexible when using standard medical terminologies for customizing controlled vocabularies, a project is ongoing towards developing a tool that can easily search and import concepts from standard vocabularies such as ICDE-9, <a href=\"https:\/\/www.limswiki.org\/index.php\/ICD-10\" title=\"ICD-10\" class=\"mw-redirect wiki-link\" target=\"_blank\" data-key=\"719f8fc71094f479738e913212228797\">ICD-10<\/a>, and SNOMED, from a local file or through NCBO BioPortal.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h3>\n<p>Although there are natural language processing tools available for extracting information from clinical reports, the majority lack the capability to support interactive feedback from human users. An interactive, online approach allows the user to coach the system using knowledge specific to the given set of reports, which may include local reporting conventions and structures. Moreover, no advanced linguistics knowledge or programming skills are required of the users; the system maintains the ordinary workflow of manual annotation systems. We perform a systematic study on the effectiveness of the online learning\u2013based method combined with controlled vocabularies for data extraction from reports with various structural patterns, and we conclude that our method is highly effective. The framework is generic and the applicability is demonstrated with diverse report types. The software will be made freely available online.<sup id=\"rdp-ebb-cite_ref-IDEAL-X_28-0\" class=\"reference\"><a href=\"#cite_note-IDEAL-X-28\" rel=\"external_link\">[28]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgments\">Acknowledgments<\/span><\/h2>\n<p>The study was funded by Center for Disease Control and Prevention, contract #200-2014-M-59415.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h2>\n<p>None declared.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Multimedia_Appendix_1\">Multimedia Appendix 1<\/span><\/h2>\n<p><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/medinform.jmir.org\/article\/downloadSuppFile\/7235\/51185\" target=\"_blank\">Test cases<\/a> (PDF file; 32KB)\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-ZhengIDEAL14-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhengIDEAL14_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Zheng, S. (07 September 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.youtube.com\/watch?v=Q-DrWi31nv0\" target=\"_blank\">\"IDEAL-X: Information and Data Extraction using Adaptive Learning\"<\/a>. <i>YouTube<\/i>. Google, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.youtube.com\/watch?v=Q-DrWi31nv0\" target=\"_blank\">https:\/\/www.youtube.com\/watch?v=Q-DrWi31nv0<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 April 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=IDEAL-X%3A+Information+and+Data+Extraction+using+Adaptive+Learning&rft.atitle=YouTube&rft.aulast=Zheng%2C+S.&rft.au=Zheng%2C+S.&rft.date=07+September+2014&rft.pub=Google%2C+Inc&rft_id=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQ-DrWi31nv0&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Shalev-ShwartzOnline07-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Shalev-ShwartzOnline07_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-Shalev-ShwartzOnline07_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Shalev-Shwartz, S. (July 2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ttic.uchicago.edu\/~shai\/papers\/ShalevThesis07.pdf\" target=\"_blank\">\"Online Learning: Theory, Algorithms, and Applications\"<\/a> (PDF). University of Chicago<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ttic.uchicago.edu\/~shai\/papers\/ShalevThesis07.pdf\" target=\"_blank\">http:\/\/ttic.uchicago.edu\/~shai\/papers\/ShalevThesis07.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 May 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Online+Learning%3A+Theory%2C+Algorithms%2C+and+Applications&rft.atitle=&rft.aulast=Shalev-Shwartz%2C+S.&rft.au=Shalev-Shwartz%2C+S.&rft.date=July+2007&rft.pub=University+of+Chicago&rft_id=http%3A%2F%2Fttic.uchicago.edu%2F%7Eshai%2Fpapers%2FShalevThesis07.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Shalev-ShwartzOnline12-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Shalev-ShwartzOnline12_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-Shalev-ShwartzOnline12_3-1\" rel=\"external_link\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shalev-Shwartz, S. (2012). \"Online Learning and Online Convex Optimization\". <i>Foundations and Trends in Machine Learning<\/i> <b>4<\/b> (2): 107\u2013194. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1561%2F2200000018\" target=\"_blank\">10.1561\/2200000018<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Online+Learning+and+Online+Convex+Optimization&rft.jtitle=Foundations+and+Trends+in+Machine+Learning&rft.aulast=Shalev-Shwartz%2C+S.&rft.au=Shalev-Shwartz%2C+S.&rft.date=2012&rft.volume=4&rft.issue=2&rft.pages=107%E2%80%93194&rft_id=info:doi\/10.1561%2F2200000018&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmaleOnline06-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmaleOnline06_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smale, S.; Yao, Y. (2006). \"Online Learning Algorithms\". <i>Foundations of Computational Mathematics<\/i> <b>6<\/b> (2): 145\u2013170. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10208-004-0160-z\" target=\"_blank\">10.1007\/s10208-004-0160-z<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Online+Learning+Algorithms&rft.jtitle=Foundations+of+Computational+Mathematics&rft.aulast=Smale%2C+S.%3B+Yao%2C+Y.&rft.au=Smale%2C+S.%3B+Yao%2C+Y.&rft.date=2006&rft.volume=6&rft.issue=2&rft.pages=145%E2%80%93170&rft_id=info:doi\/10.1007%2Fs10208-004-0160-z&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Crowley_caTIES10-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Crowley_caTIES10_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Crowley, R.S.; Castine, M.; Mitchell, K. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995710\" target=\"_blank\">\"caTIES: a grid based system for coding and retrieval of surgical pathology reports and tissue specimens in support of translational research\"<\/a>. <i>JAMIA<\/i> <b>17<\/b> (3): 253-64. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fjamia.2009.002295\" target=\"_blank\">10.1136\/jamia.2009.002295<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2995710\/\" target=\"_blank\">PMC2995710<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20442142\" target=\"_blank\">20442142<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995710\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995710<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=caTIES%3A+a+grid+based+system+for+coding+and+retrieval+of+surgical+pathology+reports+and+tissue+specimens+in+support+of+translational+research&rft.jtitle=JAMIA&rft.aulast=Crowley%2C+R.S.%3B+Castine%2C+M.%3B+Mitchell%2C+K.+et+al.&rft.au=Crowley%2C+R.S.%3B+Castine%2C+M.%3B+Mitchell%2C+K.+et+al.&rft.date=2010&rft.volume=17&rft.issue=3&rft.pages=253-64&rft_id=info:doi\/10.1136%2Fjamia.2009.002295&rft_id=info:pmc\/PMC2995710&rft_id=info:pmid\/20442142&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2995710&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XuMedEx10-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-XuMedEx10_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xu, H.; Stenner, S.P.; Doan, S. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995636\" target=\"_blank\">\"MedEx: A medication information extraction system for clinical narratives\"<\/a>. <i>JAMIA<\/i> <b>17<\/b> (1): 19\u201324. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M3378\" target=\"_blank\">10.1197\/jamia.M3378<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2995636\/\" target=\"_blank\">PMC2995636<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20064797\" target=\"_blank\">20064797<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995636\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995636<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MedEx%3A+A+medication+information+extraction+system+for+clinical+narratives&rft.jtitle=JAMIA&rft.aulast=Xu%2C+H.%3B+Stenner%2C+S.P.%3B+Doan%2C+S.+et+al.&rft.au=Xu%2C+H.%3B+Stenner%2C+S.P.%3B+Doan%2C+S.+et+al.&rft.date=2010&rft.volume=17&rft.issue=1&rft.pages=19%E2%80%9324&rft_id=info:doi\/10.1197%2Fjamia.M3378&rft_id=info:pmc\/PMC2995636&rft_id=info:pmid\/20064797&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2995636&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FriedmanAuto04-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FriedmanAuto04_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Friedman, C.; Shagina, L.; Lussier, Y.; Hripcsak, G. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC516246\" target=\"_blank\">\"Automated encoding of clinical documents based on natural language processing\"<\/a>. <i>JAMIA<\/i> <b>11<\/b> (5): 392\u2013402. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M1552\" target=\"_blank\">10.1197\/jamia.M1552<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC516246\/\" target=\"_blank\">PMC516246<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15187068\" target=\"_blank\">15187068<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC516246\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC516246<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automated+encoding+of+clinical+documents+based+on+natural+language+processing&rft.jtitle=JAMIA&rft.aulast=Friedman%2C+C.%3B+Shagina%2C+L.%3B+Lussier%2C+Y.%3B+Hripcsak%2C+G.&rft.au=Friedman%2C+C.%3B+Shagina%2C+L.%3B+Lussier%2C+Y.%3B+Hripcsak%2C+G.&rft.date=2004&rft.volume=11&rft.issue=5&rft.pages=392%E2%80%93402&rft_id=info:doi\/10.1197%2Fjamia.M1552&rft_id=info:pmc\/PMC516246&rft_id=info:pmid\/15187068&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC516246&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SavovaMayo10-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SavovaMayo10_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Savova, G.K.; Masanz, J.J.; Ogren, P.V. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995668\" target=\"_blank\">\"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): Architecture, component evaluation and applications\"<\/a>. <i>JAMIA<\/i> <b>17<\/b> (5): 507\u201313. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fjamia.2009.001560\" target=\"_blank\">10.1136\/jamia.2009.001560<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2995668\/\" target=\"_blank\">PMC2995668<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20819853\" target=\"_blank\">20819853<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995668\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995668<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Mayo+clinical+Text+Analysis+and+Knowledge+Extraction+System+%28cTAKES%29%3A+Architecture%2C+component+evaluation+and+applications&rft.jtitle=JAMIA&rft.aulast=Savova%2C+G.K.%3B+Masanz%2C+J.J.%3B+Ogren%2C+P.V.+et+al.&rft.au=Savova%2C+G.K.%3B+Masanz%2C+J.J.%3B+Ogren%2C+P.V.+et+al.&rft.date=2010&rft.volume=17&rft.issue=5&rft.pages=507%E2%80%9313&rft_id=info:doi\/10.1136%2Fjamia.2009.001560&rft_id=info:pmc\/PMC2995668&rft_id=info:pmid\/20819853&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2995668&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AronsonEffective01-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AronsonEffective01_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aronson, A.R. (2001). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2243666\" target=\"_blank\">\"Effective mapping of biomedical text to the UMLS Metathesaurus: The MetaMap program\"<\/a>. <i>Proceedings AMIA Symposium<\/i> <b>2001<\/b>: 17\u201321. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2243666\/\" target=\"_blank\">PMC2243666<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11825149\" target=\"_blank\">11825149<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2243666\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2243666<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Effective+mapping+of+biomedical+text+to+the+UMLS+Metathesaurus%3A+The+MetaMap+program&rft.jtitle=Proceedings+AMIA+Symposium&rft.aulast=Aronson%2C+A.R.&rft.au=Aronson%2C+A.R.&rft.date=2001&rft.volume=2001&rft.pages=17%E2%80%9321&rft_id=info:pmc\/PMC2243666&rft_id=info:pmid\/11825149&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2243666&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZengExtracting06-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZengExtracting06_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zeng, Q.T.; Goryachev, S.; Weiss, S. et al. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1553439\" target=\"_blank\">\"Extracting principal diagnosis, co-morbidity and smoking status for asthma research: Evaluation of a natural language processing system\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>6<\/b>: 30. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-6-30\" target=\"_blank\">10.1186\/1472-6947-6-30<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1553439\/\" target=\"_blank\">PMC1553439<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16872495\" target=\"_blank\">16872495<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1553439\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1553439<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Extracting+principal+diagnosis%2C+co-morbidity+and+smoking+status+for+asthma+research%3A+Evaluation+of+a+natural+language+processing+system&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Zeng%2C+Q.T.%3B+Goryachev%2C+S.%3B+Weiss%2C+S.+et+al.&rft.au=Zeng%2C+Q.T.%3B+Goryachev%2C+S.%3B+Weiss%2C+S.+et+al.&rft.date=2006&rft.volume=6&rft.pages=30&rft_id=info:doi\/10.1186%2F1472-6947-6-30&rft_id=info:pmc\/PMC1553439&rft_id=info:pmid\/16872495&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1553439&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HoiLIBOL14-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HoiLIBOL14_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hoi, S.C.H.; Wang, J.; Zhao, P. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.jmlr.org\/papers\/v15\/hoi14a.html\" target=\"_blank\">\"LIBOL: A Library for Online Learning Algorithms\"<\/a>. <i>Journal of Machine Learning Research<\/i> <b>15<\/b> (Feb): 495\u2013499<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.jmlr.org\/papers\/v15\/hoi14a.html\" target=\"_blank\">http:\/\/www.jmlr.org\/papers\/v15\/hoi14a.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LIBOL%3A+A+Library+for+Online+Learning+Algorithms&rft.jtitle=Journal+of+Machine+Learning+Research&rft.aulast=Hoi%2C+S.C.H.%3B+Wang%2C+J.%3B+Zhao%2C+P.&rft.au=Hoi%2C+S.C.H.%3B+Wang%2C+J.%3B+Zhao%2C+P.&rft.date=2014&rft.volume=15&rft.issue=Feb&rft.pages=495%E2%80%93499&rft_id=http%3A%2F%2Fwww.jmlr.org%2Fpapers%2Fv15%2Fhoi14a.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CardieProposal98-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CardieProposal98_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Cardie, C.; Pierce, D. (06 July 1998). \"Proposal for an Interactive Environment for Information Extraction\". <i>Cornell University Computer Science Technical Report TR98-1702<\/i>. Cornell University.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Proposal+for+an+Interactive+Environment+for+Information+Extraction&rft.atitle=Cornell+University+Computer+Science+Technical+Report+TR98-1702&rft.aulast=Cardie%2C+C.%3B+Pierce%2C+D.&rft.au=Cardie%2C+C.%3B+Pierce%2C+D.&rft.date=06+July+1998&rft.pub=Cornell+University&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CiravegnaDesigning03-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CiravegnaDesigning03_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ciravegna, F.; Wilks, Y. (2003). \"Designing Adaptive Information Extraction for the Semantic Web in Amilcare\". In Handschuh, S.; Staab, S.. <i>Annotation for the Semantic Web<\/i>. <b>96<\/b>. IOS Press. pp. 112\u2013127. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781586033453.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Designing+Adaptive+Information+Extraction+for+the+Semantic+Web+in+Amilcare&rft.atitle=Annotation+for+the+Semantic+Web&rft.aulast=Ciravegna%2C+F.%3B+Wilks%2C+Y.&rft.au=Ciravegna%2C+F.%3B+Wilks%2C+Y.&rft.date=2003&rft.volume=96&rft.pages=pp.%26nbsp%3B112%E2%80%93127&rft.pub=IOS+Press&rft.isbn=9781586033453&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CiravegnaAdaptive01-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CiravegnaAdaptive01_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-CiravegnaAdaptive01_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ciravegna, F. (2001). \"Adaptive information extraction from text by rule induction and generalisation\". <i>Proceedings of the 17th International Joint Conference on Artificial Intelligence<\/i> <b>2<\/b>: 1251\u20131256. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781558608122.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adaptive+information+extraction+from+text+by+rule+induction+and+generalisation&rft.jtitle=Proceedings+of+the+17th+International+Joint+Conference+on+Artificial+Intelligence&rft.aulast=Ciravegna%2C+F.&rft.au=Ciravegna%2C+F.&rft.date=2001&rft.volume=2&rft.pages=1251%E2%80%931256&rft.isbn=9781558608122&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GobbelAssisted14-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GobbelAssisted14_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gobbel, G.T.; Garvin, J.; Reeves, R. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4147611\" target=\"_blank\">\"Assisted annotation of medical free text using RapTAT\"<\/a>. <i>JAMIA<\/i> <b>21<\/b> (5): 833-41. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2013-002255\" target=\"_blank\">10.1136\/amiajnl-2013-002255<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4147611\/\" target=\"_blank\">PMC4147611<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24431336\" target=\"_blank\">24431336<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4147611\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4147611<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Assisted+annotation+of+medical+free+text+using+RapTAT&rft.jtitle=JAMIA&rft.aulast=Gobbel%2C+G.T.%3B+Garvin%2C+J.%3B+Reeves%2C+R.+et+al.&rft.au=Gobbel%2C+G.T.%3B+Garvin%2C+J.%3B+Reeves%2C+R.+et+al.&rft.date=2014&rft.volume=21&rft.issue=5&rft.pages=833-41&rft_id=info:doi\/10.1136%2Famiajnl-2013-002255&rft_id=info:pmc\/PMC4147611&rft_id=info:pmid\/24431336&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4147611&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SettlesActive09-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SettlesActive09_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Settles, B. (January 2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/research.cs.wisc.edu\/techreports\/2009\/TR1648.pdf\" target=\"_blank\">\"Active Learning Literature Survey\"<\/a> (PDF). University of Wisconsin<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/research.cs.wisc.edu\/techreports\/2009\/TR1648.pdf\" target=\"_blank\">https:\/\/research.cs.wisc.edu\/techreports\/2009\/TR1648.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 May 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Active+Learning+Literature+Survey&rft.atitle=&rft.aulast=Settles%2C+B.&rft.au=Settles%2C+B.&rft.date=January+2009&rft.pub=University+of+Wisconsin&rft_id=https%3A%2F%2Fresearch.cs.wisc.edu%2Ftechreports%2F2009%2FTR1648.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OlssonALit09-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OlssonALit09_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Olsson, F. (17 April 2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/soda.swedish-ict.se\/3600\/1\/SICS-T--2009-06--SE.pdf\" target=\"_blank\">\"A literature survey of active machine learning in the context of natural language processing\"<\/a> (PDF). Swedish Institute of Computer Science<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/soda.swedish-ict.se\/3600\/1\/SICS-T--2009-06--SE.pdf\" target=\"_blank\">http:\/\/soda.swedish-ict.se\/3600\/1\/SICS-T--2009-06--SE.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 May 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+literature+survey+of+active+machine+learning+in+the+context+of+natural+language+processing&rft.atitle=&rft.aulast=Olsson%2C+F.&rft.au=Olsson%2C+F.&rft.date=17+April+2009&rft.pub=Swedish+Institute+of+Computer+Science&rft_id=http%3A%2F%2Fsoda.swedish-ict.se%2F3600%2F1%2FSICS-T--2009-06--SE.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SettlesClosing11-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SettlesClosing11_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Settles, B. (2011). \"Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances\". <i>Proceedings of the Conference on Empirical Methods in Natural Language Processing<\/i> <b>2011<\/b>: 1467\u20131478. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781937284114.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Closing+the+loop%3A+Fast%2C+interactive+semi-supervised+annotation+with+queries+on+features+and+instances&rft.jtitle=Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing&rft.aulast=Settles%2C+B.&rft.au=Settles%2C+B.&rft.date=2011&rft.volume=2011&rft.pages=1467%E2%80%931478&rft.isbn=9781937284114&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChenApplyingMEDLINE13-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChenApplyingMEDLINE13_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chen, Y.; Cao, H.; Mei, Q. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3756255\" target=\"_blank\">\"Applying active learning to supervised word sense disambiguation in MEDLINE\"<\/a>. <i>JAMIA<\/i> <b>2013<\/b>: 1001\u20131006. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2012-001244\" target=\"_blank\">10.1136\/amiajnl-2012-001244<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3756255\/\" target=\"_blank\">PMC3756255<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23364851\" target=\"_blank\">23364851<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3756255\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3756255<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Applying+active+learning+to+supervised+word+sense+disambiguation+in+MEDLINE&rft.jtitle=JAMIA&rft.aulast=Chen%2C+Y.%3B+Cao%2C+H.%3B+Mei%2C+Q.+et+al.&rft.au=Chen%2C+Y.%3B+Cao%2C+H.%3B+Mei%2C+Q.+et+al.&rft.date=2013&rft.volume=2013&rft.pages=1001%E2%80%931006&rft_id=info:doi\/10.1136%2Famiajnl-2012-001244&rft_id=info:pmc\/PMC3756255&rft_id=info:pmid\/23364851&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3756255&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChenApplyingEHR13-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChenApplyingEHR13_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chen, Y.; Carroll, R.J.; Hinz, E.R. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3861916\" target=\"_blank\">\"Applying active learning to high-throughput phenotyping algorithms for electronic health records data\"<\/a>. <i>JAMIA<\/i> <b>20<\/b> (e2): e253-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2013-001945\" target=\"_blank\">10.1136\/amiajnl-2013-001945<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3861916\/\" target=\"_blank\">PMC3861916<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23851443\" target=\"_blank\">23851443<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3861916\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3861916<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Applying+active+learning+to+high-throughput+phenotyping+algorithms+for+electronic+health+records+data&rft.jtitle=JAMIA&rft.aulast=Chen%2C+Y.%3B+Carroll%2C+R.J.%3B+Hinz%2C+E.R.+et+al.&rft.au=Chen%2C+Y.%3B+Carroll%2C+R.J.%3B+Hinz%2C+E.R.+et+al.&rft.date=2013&rft.volume=20&rft.issue=e2&rft.pages=e253-9&rft_id=info:doi\/10.1136%2Famiajnl-2013-001945&rft_id=info:pmc\/PMC3861916&rft_id=info:pmid\/23851443&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3861916&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EapenAggregate13-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EapenAggregate13_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Eapen, D.J.; Manocha, P.; Patel, R.S. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4066955\" target=\"_blank\">\"Aggregate risk score based on markers of inflammation, cell stress, and coagulation is an independent predictor of adverse cardiovascular outcomes\"<\/a>. <i>Journal of the American College of Cardiology<\/i> <b>62<\/b> (4): 329-37. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jacc.2013.03.072\" target=\"_blank\">10.1016\/j.jacc.2013.03.072<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4066955\/\" target=\"_blank\">PMC4066955<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23665099\" target=\"_blank\">23665099<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4066955\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4066955<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Aggregate+risk+score+based+on+markers+of+inflammation%2C+cell+stress%2C+and+coagulation+is+an+independent+predictor+of+adverse+cardiovascular+outcomes&rft.jtitle=Journal+of+the+American+College+of+Cardiology&rft.aulast=Eapen%2C+D.J.%3B+Manocha%2C+P.%3B+Patel%2C+R.S.+et+al.&rft.au=Eapen%2C+D.J.%3B+Manocha%2C+P.%3B+Patel%2C+R.S.+et+al.&rft.date=2013&rft.volume=62&rft.issue=4&rft.pages=329-37&rft_id=info:doi\/10.1016%2Fj.jacc.2013.03.072&rft_id=info:pmc\/PMC4066955&rft_id=info:pmid\/23665099&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4066955&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OpenNLP-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OpenNLP_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/opennlp.apache.org\/\" target=\"_blank\">\"OpenNLP\"<\/a>. Apache Software Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opennlp.apache.org\/\" target=\"_blank\">http:\/\/opennlp.apache.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 April 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OpenNLP&rft.atitle=&rft.pub=Apache+Software+Foundation&rft_id=http%3A%2F%2Fopennlp.apache.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ManningIntro08-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ManningIntro08_23-0\" rel=\"external_link\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-ManningIntro08_23-1\" rel=\"external_link\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-ManningIntro08_23-2\" rel=\"external_link\">23.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Manning, C.D.; Raghavan, P.; Sch\u00fctze, H. (2008). <i>Introduction to Information Retrieval<\/i>. Cambridge University Press. pp. 506. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780521865715.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Introduction+to+Information+Retrieval&rft.aulast=Manning%2C+C.D.%3B+Raghavan%2C+P.%3B+Sch%C3%BCtze%2C+H.&rft.au=Manning%2C+C.D.%3B+Raghavan%2C+P.%3B+Sch%C3%BCtze%2C+H.&rft.date=2008&rft.pages=pp.%26nbsp%3B506&rft.pub=Cambridge+University+Press&rft.isbn=9780521865715&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElliottHidden08-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ElliottHidden08_24-0\" rel=\"external_link\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-ElliottHidden08_24-1\" rel=\"external_link\">24.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Elliott, R.J.; Aggoun, L.; Moore, J.B. (2008). <i>Hidden Markov Models: Estimation and Control<\/i>. Springer. pp. 382. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780387943640.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Hidden+Markov+Models%3A+Estimation+and+Control&rft.aulast=Elliott%2C+R.J.%3B+Aggoun%2C+L.%3B+Moore%2C+J.B.&rft.au=Elliott%2C+R.J.%3B+Aggoun%2C+L.%3B+Moore%2C+J.B.&rft.date=2008&rft.pages=pp.%26nbsp%3B382&rft.pub=Springer&rft.isbn=9780387943640&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-F.C3.BCrnkranzSeparate99-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-F.C3.BCrnkranzSeparate99_25-0\" rel=\"external_link\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-F.C3.BCrnkranzSeparate99_25-1\" rel=\"external_link\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">F\u00fcrnkranz, J. (1999). \"Separate-and-Conquer Rule Learning\". <i>Artificial Intelligence Review<\/i> <b>13<\/b> (1): 3\u201354. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1023%2FA%3A1006524209794\" target=\"_blank\">10.1023\/A:1006524209794<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Separate-and-Conquer+Rule+Learning&rft.jtitle=Artificial+Intelligence+Review&rft.aulast=F%C3%BCrnkranz%2C+J.&rft.au=F%C3%BCrnkranz%2C+J.&rft.date=1999&rft.volume=13&rft.issue=1&rft.pages=3%E2%80%9354&rft_id=info:doi\/10.1023%2FA%3A1006524209794&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuangANovel07-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HuangANovel07_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Huang, Y.; Lowe, H.J. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2244882\" target=\"_blank\">\"A novel hybrid approach to automated negation detection in clinical radiology reports\"<\/a>. <i>JAMIA<\/i> <b>14<\/b> (3): 304\u201311. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2284\" target=\"_blank\">10.1197\/jamia.M2284<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2244882\/\" target=\"_blank\">PMC2244882<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17329723\" target=\"_blank\">17329723<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2244882\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2244882<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+novel+hybrid+approach+to+automated+negation+detection+in+clinical+radiology+reports&rft.jtitle=JAMIA&rft.aulast=Huang%2C+Y.%3B+Lowe%2C+H.J.&rft.au=Huang%2C+Y.%3B+Lowe%2C+H.J.&rft.date=2007&rft.volume=14&rft.issue=3&rft.pages=304%E2%80%9311&rft_id=info:doi\/10.1197%2Fjamia.M2284&rft_id=info:pmc\/PMC2244882&rft_id=info:pmid\/17329723&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2244882&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JiangAStudy11-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JiangAStudy11_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jiang, M.; Chen, Y.; Liu, M. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3168315\" target=\"_blank\">\"A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries\"<\/a>. <i>JAMIA<\/i> <b>18<\/b> (5): 601-6. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2011-000163\" target=\"_blank\">10.1136\/amiajnl-2011-000163<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3168315\/\" target=\"_blank\">PMC3168315<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21508414\" target=\"_blank\">21508414<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3168315\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3168315<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+study+of+machine-learning-based+approaches+to+extract+clinical+entities+and+their+assertions+from+discharge+summaries&rft.jtitle=JAMIA&rft.aulast=Jiang%2C+M.%3B+Chen%2C+Y.%3B+Liu%2C+M.+et+al.&rft.au=Jiang%2C+M.%3B+Chen%2C+Y.%3B+Liu%2C+M.+et+al.&rft.date=2011&rft.volume=18&rft.issue=5&rft.pages=601-6&rft_id=info:doi\/10.1136%2Famiajnl-2011-000163&rft_id=info:pmc\/PMC3168315&rft_id=info:pmid\/21508414&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3168315&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IDEAL-X-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IDEAL-X_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.idealx.net\/home.html\" target=\"_blank\">\"IDEAL-X\"<\/a>. Emory University<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.idealx.net\/home.html\" target=\"_blank\">http:\/\/www.idealx.net\/home.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 May 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=IDEAL-X&rft.atitle=&rft.pub=Emory+University&rft_id=http%3A%2F%2Fwww.idealx.net%2Fhome.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>EMR<\/b>: electronic medical record\n<\/p><p><b>HMM<\/b>: Hidden Markov Model\n<\/p><p><b>NLP<\/b>: natural language processing\n<\/p><p><b>POS<\/b>: part of speech\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID was missing and was added to make the reference more useful. Grammar and vocabulary were cleaned up to make the article easier to read. In the \"Answer prediction\" subsection, the original referred to Figure 5, when it most likely should have referred to Figure 4; this was changed for this version.\n<\/p><p>Per the distribution agreement, the following copyright information is also being added: \n<\/p><p>\u00a9Shuai Zheng, James J Lu, Nima Ghasemzadeh, Salim S Hayek, Arshed A Quyyumi, Fusheng Wang. Originally published in JMIR Medical Informatics (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/medinform.jmir.org\" target=\"_blank\">http:\/\/medinform.jmir.org<\/a>), 09.05.2017.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181456\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.673 seconds\nReal time usage: 0.705 seconds\nPreprocessor visited node count: 22521\/1000000\nPreprocessor generated node count: 36744\/1000000\nPost\u2010expand include size: 183636\/2097152 bytes\nTemplate argument size: 57316\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 657.896 1 - -total\n 85.77% 564.308 1 - Template:Reflist\n 74.87% 492.578 28 - Template:Citation\/core\n 53.36% 351.060 18 - Template:Cite_journal\n 17.37% 114.266 7 - Template:Cite_web\n 9.29% 61.100 1 - Template:Infobox_journal_article\n 8.93% 58.734 1 - Template:Infobox\n 8.71% 57.290 3 - Template:Cite_book\n 8.51% 55.984 43 - Template:Citation\/identifier\n 5.37% 35.302 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10125-0!*!0!!en!5!* and timestamp 20181214181455 and revision id 30622\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies\">https:\/\/www.limswiki.org\/index.php\/Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","a67695148af73db7d41c4171d7c86cbc_images":["https:\/\/www.limswiki.org\/images\/7\/71\/Fig1_Zheng_JMIRMedInfo2017_5-2.png","https:\/\/www.limswiki.org\/images\/a\/af\/Fig2_Zheng_JMIRMedInfo2017_5-2.png","https:\/\/www.limswiki.org\/images\/a\/a6\/Fig3_Zheng_JMIRMedInfo2017_5-2.png","https:\/\/www.limswiki.org\/images\/5\/57\/Fig4_Zheng_JMIRMedInfo2017_5-2.png","https:\/\/www.limswiki.org\/images\/1\/11\/Fig5_Zheng_JMIRMedInfo2017_5-2.png"],"a67695148af73db7d41c4171d7c86cbc_timestamp":1544811295,"45e8147ab685e78229d6e364a84469a5_type":"article","45e8147ab685e78229d6e364a84469a5_title":"The effect of a test ordering software intervention on the prescription of unnecessary laboratory tests - A randomized controlled trial (Martins et al. 2017)","45e8147ab685e78229d6e364a84469a5_url":"https:\/\/www.limswiki.org\/index.php\/Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial","45e8147ab685e78229d6e364a84469a5_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:The effect of a test ordering software intervention on the prescription of unnecessary laboratory tests - A randomized controlled trial\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nThe effect of a test ordering software intervention on the prescription of unnecessary laboratory tests - A randomized controlled trialJournal\n \nBMC Medical Informatics and Decision MakingAuthor(s)\n \nMartins, C.M.; da Costa Teixeira, A.S.; de Azevedo. L.F.; S\u00e1, L.M.; Santos, P.A.; do Couto, M.L.; da Costa Pereira, A.M.;\r\nHespanhol, A.A.; da Costa Santos, C.M.Author affiliation(s)\n \nUniversity of PortoPrimary contact\n \nE-mail: carlosmartins20 at gmail dot comYear published\n \n2017Volume and issue\n \n17 (1)Page(s)\n \n20DOI\n \n10.1186\/s12911-017-0416-6ISSN\n \n1472-6947Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-017-0416-6Download\n \nhttps:\/\/bmcmedinformdecismak.biomedcentral.com\/track\/pdf\/10.1186\/s12911-017-0416-6 (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Methods \n\n3.1 Trial design \n3.2 Participants \n3.3 Interventions \n3.4 Independent variables \n3.5 Dependent variables \n3.6 Sample size \n3.7 Randomization \n3.8 Statistical methods \n3.9 Ethical considerations \n\n\n4 Results \n5 Discussion \n6 Conclusions \n7 Abbreviations \n8 Declarations \n\n8.1 Acknowledgements \n8.2 Funding \n8.3 Availability of data and materials \n8.4 Authors' contributions \n8.5 Competing interests \n8.6 Consent for publication \n8.7 Ethics approval and consent to participate \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nBackground: The way electronic health record and laboratory test ordering system software is designed may influence physicians\u2019 prescription. A randomized controlled trial was performed to measure the impact of a diagnostic and laboratory tests ordering system software modification.\nMethods: Participants were family physicians working and prescribing diagnostic and laboratory tests.\nThe intervention group had modified software with basic shortcut menu changes, where some tests were withdrawn or added, and with the implementation of an evidence-based clinical decision support system based on United States Preventive Services Task Force (USPSTF) recommendations. This intervention group was compared with typically used software (control group).\nThe outcomes were the number of tests prescribed from those: withdrawn from the basic menu; added to the basic menu; marked with green dots (USPSTF\u2019s grade A and B); and marked with red dots (USPSTF\u2019s grade D).\nResults: Comparing the monthly average number of tests prescribed before and after the software modification, from those tests that were withdrawn from the basic menu, the control group prescribed 33.8 tests per 100 consultations before and 30.8 after (p\u2009=\u20090075); the intervention group prescribed 31.3 before and 13.9 after (p\u2009<\u20090001). Comparing the tests prescribed between both groups during the intervention, from those tests that were withdrawn from the basic menu, the intervention group prescribed a monthly average of 14.0 vs. 29.3 tests per 100 consultations in the control group (p\u2009<\u20090.001). From those tests that are USPSTF\u2019s grade A and B, the intervention group prescribed 66.8 vs. 74.1 tests per 100 consultations in the control group (p\u2009=\u20090.070). From those tests categorized as USPSTF grade D, the intervention group prescribed an average of 9.8 vs. 11.8 tests per 100 consultations in the control group (p\u2009=\u20090.003).\nConclusions: Removing unnecessary tests from a quick shortcut menu of the diagnosis and laboratory tests ordering system had a significant impact and reduced unnecessary prescription of tests.\nThe fact that it was not possible to perform the randomization at the family physicians\u2019 level, but only on the computer servers is a limitation of our study. Future research should assess the impact of different test ordering systems during longer periods.\nTrial registration: ISRCTN45427977, May 1st 2014 (retrospectively registered).\nKeywords: Preventive health services, primary health care, evidence-based practice, decision support systems, clinical decision making, computer-assisted\n\nBackground \nInformatics has undoubtedly changed the way societies live, socialize, learn, work, and deal with healthcare. We now live in a period of increasing concern about the excessive presence of medicine in our lives.[1][2][3] When inefficient software is combined with a non-evidence-based medical practice, there is the risk of patient harm, significant impact to quality of life, and damage to the healthcare system due to unnecessary costs.\nThe implementation of electronic health records (EHRs) has both potential benefits and drawbacks.[4] Among the benefits, the prevention of medical errors and the promotion of patient safety has often been mentioned and confirmed in clinical practice.[4][5][6] Despite the positive effects of EHR implementation in clinical practice, a range of barriers faced by physicians has been identified. These barriers may include technical and financial aspects, time, psychological, social, legal, and organizational changes to the process.[7] After having removed the first barriers to EHR implementation, it is now time to implement continuing improvement and development of the available tools and to incorporate the scientific evidence obtained to this point.[4][8][9][10]\nTo achieve better patient safety standards and improve healthcare system cost-effectiveness, there has been a worldwide effort to implement an integrated EHR system with diagnostic and laboratory test ordering communication systems.[11][12][13] There have also been attempts to incorporate clinical decision support systems to further improve the quality of medicine. Prescribing diagnostic and laboratory tests is a key component of medical consultation. In the primary health care setting, tests are often ordered with preventive intentions and fulfillment of patient expectations.[14][15] There is also great uncertainty and variability among family physicians\u2019 ordering routines.[16][17][18] The effects of test ordering communication systems integrated with clinical decision support systems have been reported in various clinical practice settings. Main et al. have performed a systematic review of this topic and reported that integration of clinical decision support systems resulted in significant benefits to the prescribing process and practitioner performance outcomes in nearly two-thirds of the 24 studies that met the inclusion criteria.[19]\nIn Portugal, the use of EHR software with a diagnostic and laboratory test order communication system has been mandatory since September 2011. Most of the primary healthcare centers use software called Sistema de Apoio ao M\u00e9dico (Physician\u2019s Support System [SAM]). In the module used to order diagnostic and laboratory tests, physicians access a searchable test menu by two possible strategies: 1) typing the test name in a search box or 2) browsing by a shortcut menu structure (Fig. 1). Different menus are available for most areas of medicine, including basic, allergology, andrology, cardiovascular, central nervous system, digestive, dosing, endocrinology, gynecology, hematology, infectious diseases, nephrology, obstetrics, oncology, otorhinolaryngology, osteoarticular, preoperative, respiratory, rheumatology, and urology. Under each menu there is a set of specific lab tests. Physicians can choose one or more tests by double-clicking each test or can choose the entire set by double-clicking on the shortcut menu\u2019s title. For example, the basic menu is composed of uric acid, total cholesterol, creatinine, gamma-glutamyl transferase, glucose, hemogram, serum protein electrophoresis, aspartate aminotransferase, urine type 2, sedimentation rate, electrocardiogram, and lung x-ray tests.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Usual ordering communication system: the basic shortcut menu\n\n\n\nIf a physician would like to choose only hemogram and glucose, they must double-click on each test. However, if they would like the entire basic set of tests, they must double-click on the basic menu title. Our research team suspected that this basic shortcut menu is often selected during routine consultations in which patients ask for routine check-ups without specific reason. As we have shown in a previous study, there is a high prevalence of Portuguese adults (99.2%) that believe they should have routine blood and urine tests annually.[14] This statistic demonstrates the importance of examining the effectiveness and efficiency of this basic sub-menu.\nThrough a randomized controlled trial, the primary aim of the present study was to compare the effects of modifying the EHR ordering communication system (modified SAM) by changing the basic shortcut menu and adding a clinical decision support system based on the integration of the United States Preventive Services Task Force (USPSTF) recommendations. After the last primary healthcare reform in Portugal, primary healthcare centers have been divided into healthcare center groups.[20] In a healthcare center group, the informatics network is linked through servers that may serve more than one healthcare center. Creating a modified version of the SAM software requires it to be installed at a server level, which determines that all physicians at all healthcare centers served by that server will receive the same version of the software. For this reason, it was not possible to randomize the study at the physician level. Rather, we had to randomize the servers at the healthcare center group level.\n\nMethods \nTrial design \nAll servers of the Western Oporto grouping of health centers counted for randomization, except that serving the center where study authors worked (to avoid possible bias). The remaining nine servers were randomized into two groups: 1) five servers were randomly allocated to the intervention group and 2) four servers to the control group.\n\nParticipants \nAll family physicians working and prescribing diagnostic and laboratory tests in the Western Oporto group of health centers (except in those where the authors worked) participated in this study.\nData of the diagnostic and laboratory tests prescriptions were centrally collected by informatics staff belonging to the Ministry of Health and sent to the research team without patients\u2019 or physicians\u2019 identifications.\n\nInterventions \nThe control group continued to use the standard version of the EHR software (SAM) as presented in Fig. 1. The intervention group used a modified version of the software (SAM modified) installed in each server (Fig. 2). Software modifications consisted of two principal changes: \n1. Basic shortcut menu changes: The composition of the basic menu set of diagnostic laboratory tests was changed. Some tests were removed, including uric acid, serum protein electrophoresis, sedimentation rate, electrocardiogram, and lung x-ray. Other tests were added, including HDL cholesterol, fecal occult blood test, triglycerides, Pap smear, and mammography. Although some tests were removed from the basic menu, physicians were still able to request them by typing their names in the search for tests box.\n2. Addition of an evidence-based decision support: For the tests listed in Table 1, we added traffic light-based colored dots according to the USPSTF recommendations and an additional information box containing the summary of the USPSTF recommendation and a link to the integral recommendation at the USPSTF website (Fig. 2).[21]\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Modified ordering communication system: the basic shortcut menu. Red numbers: 1 - Traffic lights colored dots according to United States Preventive Services Task Force recommendations grades. 2 - Text box with the summary of the recommendation for each selected test. 3 - Link to the original recommendation at the USPSTF\u2019s website. 4 - Search for tests box, where any test, including those removed from the basic menu, can be searched and requested by typing the test\u2019s name. 5 - Legend of the colored dots\n\n\n\n\n\n\n\n\n\nTable 1. United States Preventives Services Task Force (USPSTF) recommendations (March, 2012)\r\na USPSTF grades A and B: the USPSTF recommends the service, marked with green dots. Grade D: the USPSTF recommends against the service, marked with red dots. Grade I: the USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of the service, marked with yellow dots.\r\n\nb DXA Dual-energy X-ray absorptiometry\n\n\n\nColored dots\n\nGradea\n\nTest\n\nSummary of the recommendations\n\n\nRed\n\nD\n\nPelvic ultrasound\n\nNot recommended as routine screening for ovarian cancer\n\n\nRed\n\nD\n\nCancer antigen 19-9\n\nNot recommended as routine screening for pancreatic cancer\n\n\nRed\n\nD\n\nRest electrocardiography\n\nNot recommended in asymptomatic adults at low risk for coronary heart disease events\n\n\nRed\n\nD\n\nExercise electrocardiography\n\nNot recommended in asymptomatic adults at low risk for coronary heart disease events\n\n\nRed\n\nD\n\nCarotid artery ultrasound\n\nNot recommended as screening for asymptomatic carotid artery stenosis in the general adult population\n\n\nRed\n\nD\n\nSpirometry\n\nNot recommended as screening adults for chronic obstructive pulmonary disease\n\n\nRed\n\nD\n\nHepatitis B surface antigen\n\nNot recommended as routine screening\n\n\nRed\n\nD\n\nHepatitis C antibodies\n\nNot recommended as routine screening\n\n\nYellow\n\nI\n\nTriglycerides\n\nThere is currently insufficient evidence of the benefit of including triglycerides as a part of the initial tests used to screen routinely for dyslipidemia.\n\n\nYellow\n\nI\n\nProstate-specific antigen\n\nThe evidence is insufficient to recommend for or against routine screening for prostate cancer in men younger than 75.\n\n\nYellow\n\nI\n\nLung computed tomography\n\nThe evidence is insufficient to recommend for or against routine screening for lung cancer.\n\n\nYellow\n\nI\n\nLung x-ray\n\nThe evidence is insufficient to recommend for or against routine screening for lung cancer.\n\n\nYellow\n\nI\n\nThyroid-stimulating hormone\n\nThe evidence is insufficient to recommend for or against routine screening.\n\n\nGreen\n\nB\n\nGlucose\n\nScreening for type 2 diabetes recommended in asymptomatic adults with sustained blood pressure (either treated or untreated) greater than 135\/80 mm Hg\n\n\nGreen\n\nA\/B\n\nTotal cholesterol\n\nRecommended every 5 years, for men aged 35 and older and women 45 and older if they are at increased risk for coronary heart disease, at younger ages if they are at increased risk for coronary heart disease.\n\n\nGreen\n\nB\n\nMammography\n\nBiennial screening recommended for women aged 50 to 74 years\n\n\nGreen\n\nA\n\nCervicovaginal cytology\n\nEvery 3 years screening recommended for women who have cervix, 21 to 65 years old\n\n\nGreen\n\nA\n\nFecal occult blood test\n\nRecommended annually as a possible method of screening for colorectal cancer, 50 to 75 years\n\n\nGreen\n\nA\n\nColonoscopy\n\nRecommended every 10 years as a possible method of screening for colorectal cancer, 50 to 75 years\n\n\nGreen\n\nA\n\nFlexible sigmoidoscopy\n\nRecommended every 5 years as a possible method of screening for colorectal cancer, 50 to 75 years\n\n\nGreen\n\nA\/B\n\nHDL cholesterol\n\nRecommended every 5 years, for men aged 35 and older and women 45 and older if they are at increased risk for coronary heart disease, at younger ages if they are at increased risk for coronary heart disease.\n\n\nGreen\n\nB\n\nDXAb of the hip and lumbar spine\n\nScreening recommended for osteoporosis in women aged 65 years or older and in younger women whose fracture risk is equal to or greater than that of a 65-year-old\n\n\nGreen\n\nA\n\nVenereal Disease Research Laboratory\n\nRecommended for persons at increased risk for syphilis infection\n\n\n\nThe selection of the specific tests to be removed from or added to the basic shortcut menu was made after discussion and debate with eight family physicians selected by proximity and convenience. The main rational was to remove tests that are not recommended to be used as routine screening tools and to substitute them with others that may be recommended for certain risk groups. However, creating a change that was not too disruptive to the current family physicians\u2019 practices was a concern, and that is why tests such as creatinine, gamma-glutamyl transferase, hemogram, aspartate aminotransferase, and urine type 2 were left in the basic shortcut menu.\nEHR software modifications were implemented on May 30\u201331, 2012 in all of the intervention groups\u2019 servers. Prospective monthly monitoring and data collection occurred until January 31, 2013. To allow a pre-post analysis in both groups, a retrospective monthly data collection of both control and intervention groups was also performed between December 1, 2011 and May 31, 2012.\nThe collected data included monthly parameters, including numbers of prescribing family physicians, face-to-face consultations made, and number of each prescribed diagnostic and laboratory test.\n\nIndependent variables \nGroup (control and intervention) and time (before and after the modifications) were the independent variables.\n\nDependent variables \nPrimary outcomes were chosen to assess the impact of our intervention on the number of diagnostic and laboratory tests prescribed by physicians using four different perspectives: 1) impact on the number of the prescriptions of diagnostic and laboratory tests that were withdrawn from the basic menu; 2) impact on the number of the prescriptions of diagnostic and laboratory tests that were added to the basic menu; 3) impact on the number of the prescriptions of diagnostic and laboratory tests that were marked with green dots (USPSTF recommendation grades A and B); and 4) impact on the number of prescriptions of diagnostic and laboratory tests that were marked with red dots (USPSTF recommendation grade D). Diagnostic and laboratory tests were the dependent variables.\n\nSample size \nGiven administrative and technical barriers related to the permission process for the introduction of EHR software modifications, our sample size was obtained by convenience. We decided to run this trial at the Western Oporto grouping of healthcare centers, which includes a total of 15 primary care healthcare centers and an informatics network of 10 servers. One of the servers, which served the healthcare center in which some of the study authors worked, was excluded from the study.\nThe power calculation was done for the mean number of monthly prescribed tests per 100 consultations comparison between control and intervention groups. A significance level of 0.05, a power of 0.8, and a standard deviation of two tests prescribed per 100 consultations per month was considered. A mean difference of 3.3 tests prescribed per 100 consultations per month between control and intervention groups was used.\n\nRandomization \nThe remaining nine servers were sequentially numerated and randomly assigned to the intervention and control group by an investigator blinded to the server identification. The allocation sequence was computer generated resulting in five servers (seven healthcare centers, 58 family physicians) allocated to the intervention group, and four servers (seven healthcare centers, 59 family physicians) allocated to the control group.\nTo guarantee allocation anonymity, family physicians at each healthcare center only received information about this trial implementation after randomization has been performed. Given the nature of this trial, no consent at physician level was obtained. Consent was obtained from the Northern Regional Health Administration and the Executive Council of the Western Oporto Group of Health Centers.\n\nStatistical methods \nTo examine whether the software modification changed test prescription trends we performed an interrupted time series analysis with an autoregressive integrated moving average (ARIMA) model, using the monthly number of tests prescribed per 100 consultations and the intervention (the software modification) as a dichotomous variable (before and after intervention). In the control group, we also performed an ARIMA model analysis using the monthly number of tests prescribed per 100 consultations and the same dichotomous variable (before and after intervention), although there was no intervention in this group. The before and after intervention comparison of the monthly average number of diagnostic and lab tests prescribed per 100 consultations between control and intervention groups was made using an independent sample t test. The Bonferroni correction was used for adjusting for multiple testing (several diagnostic and laboratory tests).[22] A significance level of 0.05 was used. As size effect measure, Cohen\u2019s d was calculated and a d of \u22650.8 was considered a large effect.\n\nEthical considerations \nThis study was approved by the Northern Regional Health Administration Medical Ethics Committee with several considerations: \u201c1) Given that the study will collect only anonymous data there is no need for the obtainment of informed consent by physicians; 2) The study is of great relevance and with an expected practical interest of the results; and 3) The methodology used safeguards the rights of participants.\u201d\nIn accordance with the Medical Ethics Committee, before the implementation of the study (but after the randomized allocation), a letter was sent to all Western Oporto Group of healthcare centers informing them of and explaining the study aims and methodology.\n\nResults \nThe number of servers enrolled was constant without loss (Fig. 3). The number of family physicians after the intervention ranged from 58 to 64 in the intervention group and from 55 to 70 in the control group. Recruitment was performed in April 2012, and the trial began on June 1, 2012, with a follow-up period until January 31, 2013 (eight month follow-up). Baseline data are presented in Table 2.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Flowchart summary of the trial\n\n\n\n\n\n\n\n\n\nTable 2. Baseline data ranges for each group based on the last six months before the electronic health record (EHR) software modification\n\n\n\n\nUsual software (control group) min - max\n\nModified software (intervention group) min - max\n\n\nFamily physicians\n\n57\u201361\n\n55\u201363\n\n\nTotal of patients covered by health centers\n\n110,781\u2013110,783\n\n79,847\u201379,850\n\n\nFace-to-face consultations\n\n10,725\u201313,493\n\n10,047\u201312,598\n\n\n\nFor both groups, Table 3 compares the monthly average number of diagnostic and laboratory tests prescribed for each 100 consultations before and after EHR software modification, under a time series analysis perspective (ARIMA model). Within the five tests that were withdrawn from the basic menu, we observe a statistically significant reduction in the prescription of four tests (uric acid, serum protein electrophoresis, sedimentation rate, and lung x-ray tests) in the intervention group after software modification. Within the five tests that were added to the basic menu, we observed a significant increase in prescription of the fecal occult blood test in the intervention group after software modification. Within the tests that were marked with green dots (USPSTF recommendation grades A and B) there were no significant increases in either the control or intervention groups. Within the tests that were marked with red dots (USPSTF recommendation grade D), there was a significant reduction only in the prescription of cancer antigen 19-9, but the amount of this test that was prescribed was so low that this variation may be justified by other factors and not a result of software modification.\n\r\n\n\n\n\n\n\n\n\nTable 3. Pre-post comparison of the monthly number of diagnostic and lab tests prescribed per 100 consultations (time series analysis, ARIMA model)\r\n\na Tests that were also marked with red dots\r\n\nb Tests that were also marked with green dots\r\n\n\u03b1 DXA: Dual-energy X-ray absorptiometry\r\n\n\u03b2 VDRL: Venereal Disease Research Laboratory\n\n\n\n\n\nControl group (always exposed to usual EHR software)\n\nIntervention group\n\n\nBefore\n\nAfter\n\np\n\nBefore software modification\n\nAfter software modification\n\np\n\n\nDiagnostic and lab tests that were withdrawn from the basic menu\n\n\nUric acid\n\n12.0\n\n10.9\n\n0.094\n\n11.2\n\n4.5\n\n<0.001\n\n\nSerum protein electrophoresis\n\n4.1\n\n3.2\n\n0.001\n\n4.7\n\n0.6\n\n<0.001\n\n\nSedimentation rate\n\n8.3\n\n7.7\n\n0.259\n\n7.1\n\n2.0\n\n<0.001\n\n\nRest electrocardiographya\n\n7.7\n\n6.8\n\n0.189\n\n6.4\n\n5.4\n\n0.072\n\n\nLung x-ray\n\n1.7\n\n1.6\n\n0.351\n\n1.9\n\n1.4\n\n0.002\n\n\nAll tests withdrawn\n\n33.8\n\n30.3\n\n0.075\n\n31.3\n\n13.9\n\n<0.001\n\n\nDiagnostic and laboratory tests that were added to the basic menu\n\n\nHDL cholesterolb\n\n19.0\n\n17.8\n\n0.357\n\n16.7\n\n16.0\n\n0.566\n\n\nTriglycerides\n\n19.6\n\n18.3\n\n0.349\n\n17.4\n\n16.2\n\n0.372\n\n\nFecal occult blood testb\n\n8.5\n\n7.3\n\n0.236\n\n4.9\n\n6.3\n\n0.017\n\n\nCervicovaginal cytologyb\n\n4.8\n\n4.7\n\n0.779\n\n4.5\n\n4.4\n\n0.905\n\n\nMammographyb\n\n1.9\n\n1.8\n\n0.332\n\n1.6\n\n1.7\n\n0.631\n\n\nAll tests added\n\n53.7\n\n49.9\n\n0.334\n\n45.0\n\n44.7\n\n0.909\n\n\nDiagnostic and laboratory tests that were marked with green dots (USPSTF grade A and B)\n\n\nGlucose\n\n22.7\n\n21.5\n\n0.402\n\n20.4\n\n18.8\n\n0.199\n\n\nTotal cholesterol\n\n20.6\n\n19.3\n\n0.345\n\n18.3\n\n16.9\n\n0.264\n\n\nColonoscopy\n\n1.4\n\n1.5\n\n0.325\n\n1.2\n\n1.2\n\n0.940\n\n\nFlexible sigmoidoscopy\n\n0.1\n\n0.1\n\n0.031\n\n0.1\n\n0.1\n\n0.115\n\n\nDXA\u03b1\n\n1.0\n\n0.7\n\n0.035\n\n0.6\n\n0.6\n\n0.562\n\n\nVDRL\u03b2\n\n2.1\n\n2.0\n\n0.234\n\n1.9\n\n1.6\n\n0.003\n\n\nAll green marked\n\n82.1\n\n76.7\n\n0.331\n\n70.2\n\n67.5\n\n0.560\n\n\nDiagnostic and laboratory tests that were marked with red dots (USPSTF grade D)\n\n\nPelvic ultrasound\n\n1.2\n\n1.1\n\n0.033\n\n1.0\n\n1.1\n\n0.815\n\n\nCancer antigen 19-9\n\n0.2\n\n0.2\n\n0.911\n\n0.1\n\n0.1\n\n0.013\n\n\nExercise electrocardiography\n\n0.5\n\n0.6\n\n0.308\n\n0.5\n\n0.4\n\n0.113\n\n\nCarotid artery ultrasound\n\n0.3\n\n0.2\n\n0.484\n\n0.2\n\n0.2\n\n0.072\n\n\nSpirometry\n\n0.5\n\n0.5\n\n0.959\n\n0.3\n\n0.5\n\n0.010\n\n\nHepatitis B surface antigen\n\n1.6\n\n1.7\n\n0.627\n\n1.3\n\n1.2\n\n0.147\n\n\nHepatitis C antibodies\n\n1.2\n\n1.2\n\n0.379\n\n0.9\n\n0.9\n\n0.898\n\n\nAll red marked\n\n13.1\n\n12.3\n\n0.341\n\n10.9\n\n9.7\n\n0.102\n\n\n\nTable 4 presents a direct comparison between the control and intervention groups before and after software modifications (excluding the first month following modification that was considered as a washout period). Regarding the set of tests that were withdrawn from the basic menu, we observe that there were no significant differences between either group before software modification, but that there was a significantly lower prescription rate in the intervention group for the following tests after software modification: 1) uric acid; 2) serum protein electrophoresis; and 3) sedimentation rate. When we consider all tests belonging to the set withdrawn from the basic menu, we have verified that after software modification the intervention group prescribed less than half of those tests when compared with the control group (14.01 tests per 100 consultations versus 29.29 tests per 100 consultations, p\u2009<\u20090.001). Regarding the five tests that were added to the basic menu, we observed no significant differences after software modification between the two groups. Considering the tests that were marked with green dots (USPSTF recommendation grades A and B) we observed no significantly higher prescription rates in the intervention group after the software modification compared to the control group. For the tests that were marked with red dots (USPSTF recommendation grade D), we observed significant lower prescription rates in five tests (cancer antigen 19-9, exercise electrocardiography, carotid artery ultrasound, hepatitis B surface antigen, and hepatitis C antibodies) in the intervention group after software modification.\n\r\n\n\n\n\n\n\n\n\nTable 4. Comparison of the monthly number of diagnostic and lab tests prescribed per 100 consultations between control and intervention groups, before and after EHR software modification (independent sample t test with Bonferroni correction)\na Tests that were also marked with red dots\r\n\nb Tests that were also marked with green dots\r\n\n\u03b1 DXA: Dual-energy X-ray absorptiometry\r\n\n\u03b2 VDRL: Venereal Disease Research Laboratory\n\n\n\n\n\nBefore (always exposed to usual EHR software)\n\nAfter software modification (first month, washout period, excluded)\n\n\nControl\n\nIntervention\n\np\n\nCohen's d\n\nControl\n\nIntervention\n\np\n\nCohen's d\n\n\nDiagnostic and lab tests that were withdrawn from the basic menu\n\n\nUric acid\n\n12.0\n\n11.2\n\n1.000\n\n0.632\n\n10.5\n\n4.6\n\n<0.001\n\n6.138\n\n\nSerum protein electrophoresis\n\n4.1\n\n4.7\n\n0.605\n\n-1.039\n\n3.2\n\n0.6\n\n<0.001\n\n7.384\n\n\nSedimentation rate\n\n8.3\n\n7.1\n\n0.315\n\n1.207\n\n7.5\n\n2.0\n\n<0.001\n\n10.818\n\n\nRest electrocardiographya\n\n7.7\n\n6.4\n\n0.465\n\n1.073\n\n6.6\n\n5.4\n\n0.070\n\n1.544\n\n\nLung x-ray\n\n1.7\n\n1.9\n\n0.985\n\n-0.802\n\n1.5\n\n1.4\n\n1.000\n\n0.441\n\n\nAll tests withdrawn\n\n33.8\n\n31.3\n\n0.292\n\n0.641\n\n29.29\n\n14.01\n\n<0.001\n\n7.174\n\n\nDiagnostic and laboratory tests that were added to the basic menu\n\n\nHDL cholesterolb\n\n19.0\n\n16.7\n\n0.640\n\n0.958\n\n17.2\n\n15.9\n\n1.000\n\n0.702\n\n\nTriglycerides\n\n19.6\n\n17.4\n\n0.830\n\n0.862\n\n17.7\n\n16.1\n\n0.640\n\n0.874\n\n\nFecal occult blood testb\n\n8.5\n\n4.9\n\n0.015\n\n2.207\n\n6.9\n\n6.2\n\n0.425\n\n1.003\n\n\nCervicovaginal cytologyb\n\n4.8\n\n4.5\n\n1.000\n\n0.468\n\n4.5\n\n4.4\n\n1.000\n\n0.361\n\n\nMammographyb\n\n1.9\n\n1.6\n\n0.855\n\n0.851\n\n1.7\n\n1.7\n\n1.000\n\n0.157\n\n\nAll tests added\n\n53.7\n\n45.0\n\n0.070\n\n1.172\n\n48.0\n\n44.3\n\n0.158\n\n0.805\n\n\nDiagnostic and laboratory tests that were marked with green dots (USPSTF grade A and B)\n\n\nGlucose\n\n22.7\n\n20.4\n\n0.864\n\n0.914\n\n20.9\n\n18.6\n\n0.228\n\n1.246\n\n\nTotal cholesterol\n\n20.6\n\n18.3\n\n0.930\n\n0.885\n\n18.7\n\n16.8\n\n0.450\n\n1.040\n\n\nColonoscopy\n\n1.4\n\n1.2\n\n0.042\n\n1.943\n\n1.5\n\n1.1\n\n0.006\n\n2.345\n\n\nFlexible sigmoidoscopy\n\n0.12\n\n0.07\n\n<0.001\n\n3.002\n\n0.05\n\n0.08\n\n0.186\n\n-1.326\n\n\nDXA\u03b1\n\n1.0\n\n0.6\n\n0.036\n\n2.010\n\n0.7\n\n0.5\n\n0.414\n\n1.063\n\n\nVDRL\u03b2\n\n2.1\n\n1.9\n\n0.552\n\n1.077\n\n1.9\n\n1.6\n\n0.102\n\n1.483\n\n\nAll green marked\n\n82.1\n\n70.2\n\n0.070\n\n1.170\n\n74.1\n\n66.8\n\n0.070\n\n1.064\n\n\nDiagnostic and laboratory tests that were marked with red dots (USPSTF grade D)\n\n\nPelvic ultrasound\n\n1.20\n\n1.04\n\n0.084\n\n1.774\n\n1.02\n\n1.08\n\n1.000\n\n-0.305\n\n\nCancer antigen 19-9\n\n0.18\n\n0.13\n\n0.616\n\n1.093\n\n0.17\n\n0.08\n\n<0.001\n\n2.513\n\n\nExercise electrocardiography\n\n0.52\n\n0.48\n\n1.000\n\n0.416\n\n0.56\n\n0.38\n\n0.028\n\n1.900\n\n\nCarotid artery ultrasound\n\n0.27\n\n0.23\n\n1.000\n\n0.750\n\n0.25\n\n0.15\n\n0.021\n\n1.978\n\n\nSpirometry\n\n0.47\n\n0.34\n\n0.021\n\n2.307\n\n0.45\n\n0.52\n\n0.812\n\n-0.902\n\n\nHepatitis B surface antigen\n\n1.60\n\n1.34\n\n0.084\n\n1.756\n\n1.59\n\n1.24\n\n0.028\n\n1.925\n\n\nHepatitis C antibodies\n\n1.15\n\n0.91\n\n0.021\n\n2.207\n\n1.18\n\n0.91\n\n0.014\n\n2.045\n\n\nAll red marked\n\n13.1\n\n10.9\n\n0.021\n\n1.583\n\n11.8\n\n9.8\n\n0.003\n\n1.941\n\n\n\nDiscussion \nOur results show that removing unnecessary tests from a quick shortcut menu of diagnostic and laboratory tests available on the EHR test ordering system might have significantly impacted ordering habits and reduced unnecessary test prescriptions. This result emphasized the importance of careful attention and scientific rigor when building shortcut menus in ordering systems. This impact was significant, and it could be observed from two perspectives of analysis: 1) when we compared the results for the intervention group before and after the modification of the software and 2) when we compared the intervention group with the control group.\nFurthermore, our results demonstrated that the introduction of an evidence-based decision support tool had no significant impact on the test prescription profiles. However, it should be noted that as a passive instrument, the decision for support addition did not interfere with the normal flow of the ordering system. The doctors had only visual contact with the colored recommendations and optional access to the additional information from the USPSTF recommendations. The absence of impact of this support decision tool may also be related to a high level of agreement between the way Portuguese family doctors perform preventive health services and the USPSTF recommendations as shown in a previous national cross-sectional study.[22]\nLimitations of our study include the sampling method and sample size. The moderate size of the sample might have limited our conclusions regarding the less frequently prescribed tests, but this is unlikely to have affected the main conclusion of our study.\nThe fact that it was not possible to perform randomization at the family physician level, but only at that of the computer servers, also constitutes a limitation and a possible biasing factor, since the number of servers in the Western Oporto Grouping of healthcare centers was small. In the Western Oporto Grouping of healthcare centers we had nine servers. Since our intervention was a software modification, we had to randomize at the level of the electronic server. One hypothesis would be used to consider each server as a cluster, but the problem is that some servers serve only one health unit, not being a real cluster, while others serve two or three healthcare units. The ideal would be to consider each healthcare unit as a cluster of family physicians. For that, however, we had to have access to the number of tests prescribed by each family physician, which we did not. The Ministry of Health only allowed us access to the tests prescribed by each healthcare unit. This is the reason why it is not possible to consider this study as a cluster randomized controlled trial. Without the number of tests prescribed by each family physician, we were not able to calculate the intracluster correlation coefficient.\nAnother limitation of our study is that the time available for its completion was relatively short. An extended study period could have allowed us to draw more extensive conclusions (for example, regarding the tests that are prescribed less frequently). A prolonged study period would also have diluted possible seasonal effects that might exist in seeking medical consultations and also minimized the impact of factors inherent in the economic situation of Portugal during the study period. In the context of the economic crisis, there was an increase in user fees paid by patients for medical consultations and laboratory tests in addition to Ministry of Health publications of clinical guidelines for prescriptions of some diagnostic and laboratory tests. These facts may have contributed to the gradual decrease in test prescriptions that we have observed during the study period both in the control and intervention groups. The extension of the study might also have diluted this effect and could have allowed a clearer reading of the impact of software modifications. However, the socio-economic status of intervention and control medical centers was identical. All healthcare centers involved in the study are located in the urban area of Oporto in a geographic area of approximately 22Km2.[23] All study physicians were exposed to the same Ministry of Health guidelines and general education materials.\nThese limitations were mainly due to technical issues related to the fact that our research could not disrupt the normal functioning of resources and services of the Ministry of Health. However, we consider it unlikely that any of these limitations limited the main conclusion of this study: the considerable impact of unnecessary test removal in the shortcut menu of the ordering system.\nConsidering the potential generalizability of our results to other groups of healthcare centers of the Portuguese National Health Service, we believe that there is a high probability that the same effect will be found with similar software intervention. We believe that this is the case for two reasons: 1) the size of the obtained effect was evident and 2) the main features of the ordering system of other groups of healthcare centers are similar.\nFor this study, we used the recommendations of the USPSTF as our standard guidelines because it bases its recommendations on freely accessible, evidence-based systematic reviews with recognized methodological quality that covers a considerable range of topics. Portugal is in a phase of transition regarding clinical guidelines and recommendations. The Ministry of Health has recently published guidelines covering some topics related with test prescriptions. However, many of these guidelines are still in the public discussion phase and are not based on systematic reviews but rather on experts' opinion. For these reasons we chose the USPSTF recommendations. Exceptions to this are the recommendations of the Portuguese Ministry of Health for three tests: 1) breast cancer screening by mammography every two years for women 50\u201369 years old; 2) colorectal cancer screening by a fecal occult blood test every one to two years for adults 50\u201374 years old; and 3) cervical cancer screening with a cervicovaginal cytology for women between 25 and 60 years every three years after two annual normal tests.[24] These recommendations are in line with those from USPSTF but could have influenced our results.\nWhen designing this trial, the researchers also considered the possibility of causing some negative effects as a result of the intervention. There could be two main type of negative effects: 1) on physicians because they could not find the tests in the basic shortcut menu as they previously had, and it would consume more consultation time to type the name in the search box to order one of the removed tests and 2) on patients because they could be deprived of some necessary tests. After debating this issue with some family physicians, these potential negative effects were considered negligible, mainly because there was no interference with physicians' freedom to prescribe any medical test, and the use of the search box as an alternative to the shortcut menu was not considered a major obstacle.\nThe reduction of unnecessary test prescriptions that we verified was in line with other studies in which changes in laboratory request forms also resulted in the reduction of unnecessary tests requests.[25][26][27] However, there are some practical implications that may result from this study. For example, the Portuguese Health Authorities could use these results to improve test ordering software and to reduce the prescription of unnecessary tests at a national level. This study also reinforces the importance of adequately testing medical related software tools, and we hope that in the future this will be the rule for other medical software tools.\n\nConclusions \nOur results might have a significant impact on improving the design of shortcut menus of diagnostic and laboratory test ordering systems either in the Portuguese National Health Service or in other countries\u2019 health systems. These improvements can help reduce the prescription of unnecessary tests, leading to the reduction of negative patient effects and to the reduction of unnecessary costs. These study results demonstrate the importance of testing and evaluating various aspects of medical informatics programs to improve efficiency and contribute to improved clinical practice and clinical outcomes.\n\nAbbreviations \nARIMA: Autoregressive integrated moving average\nEHR: Electronic health records\nSAM: Sistema de Apoio ao M\u00e9dico (Physician\u2019s Support System)\nUSPSTF: United States Preventive Services Task Force\n\nDeclarations \nAcknowledgements \nNot applicable.\n\nFunding \nWe have received financial support of 15,000 Eur from the Astrazeneca Portugal Foundation\u2019s 2010 program for support of research. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The co-author Andreia Teixeira thanks the financial support by FCT (SFRH\/BPD\/86383\/2012).\n\nAvailability of data and materials \nThe data are not available because the ethics committee only provided authorization for the research team to perform the raw data analysis.\n\nAuthors' contributions \nCMSM had the original idea of the study, designed the trial and made the quality control during the data collection, drafted and revised the paper. LFRA contributted to the trial design and revised the paper. ASCT and CMNCS planned, supervised and has performed the statistical data analysis, contributed to the paper draft and revised the paper. LMBS, PAAPS, MLGDC, AMRCP and AAOPH revised the paper. All the authors read and approved the final version of the manuscript.\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nConsent for publication \nNot applicable.\n\nEthics approval and consent to participate \nThis study was approved by the Northern Regional Health Administration Medical Ethics Committee that considered: a) Given that the study will collect only anonymous data there is no need for the obtainment of informed consent by physicians; b) The study is of great relevance and with an expected practical interest of the results; and c) the methodology used safeguards the rights of participants.\nIn accordance with the Medical Ethics Committee, before the implementation of the study (but after the randomized allocation), a letter was sent to all Western Oporto Group of health centers informing and explaining the aims and methodology of the study.\n\nReferences \n\n\n\u2191 Glasziou, P.; Moynihan, R.; Richards, T. et al. (2013). \"Too much medicine; too little care\". BMJ 347: f4247. doi:10.1136\/bmj.f4247. PMID 23820022.   \n\n\u2191 Moynihan, R.; Doust, J.; Henry, D. (2012). \"Preventing overdiagnosis: How to stop harming the healthy\". BMJ 344: e3502. doi:10.1136\/bmj.e3502. PMID 22645185.   \n\n\u2191 Getz, L.; Sigurdsson, J.A.; Hetlevik, I. (2003). \"Is opportunistic disease prevention in the consultation ethically justifiable?\". BMJ 327 (7413): 498\u2013500. doi:10.1136\/bmj.327.7413.498. PMC PMC188390. PMID 12946974. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC188390 .   \n\n\u2191 4.0 4.1 4.2 Raposo, V.L. (2015). \"Electronic health records: Is it a risk worth taking in healthcare delivery?\". GMS Health Technology Assessment 11: Doc02. doi:10.3205\/hta000123. PMC PMC4677576. PMID 26693253. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4677576 .   \n\n\u2191 Ben-Assuli, O. (2015). \"Electronic health records, adoption, quality of care, legal and privacy issues and their implementation in emergency departments\". Health Policy 119 (3): 287\u201397. doi:10.1016\/j.healthpol.2014.11.014. PMID 25483873.   \n\n\u2191 Ben-Assuli, O.; Leshno, M. (2016). \"Assessing electronic health record systems in emergency departments: Using a decision analytic Bayesian model\". Health Informatics Journal 22 (3): 712\u201329. doi:10.1177\/1460458215584203. PMID 26033468.   \n\n\u2191 Boonstra, A.; Broekhuis, M. (2010). \"Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions\". BMC Health Services Research 10: 231. doi:10.1186\/1472-6963-10-231. PMC PMC2924334. PMID 20691097. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2924334 .   \n\n\u2191 Roukema, J.; Los, R.K.; Bleeker, S.E. (2006). \"Paper versus computer: Feasibility of an electronic medical record in general pediatrics\". Pediatrics 117 (1): 15\u201321. doi:10.1542\/peds.2004-2741. PMID 16396855.   \n\n\u2191 Bowman, S. (2013). \"Impact of electronic health record systems on information integrity: Quality and safety implications\". Perspectives in Health Information Management 10: 1c. PMC PMC3797550. PMID 24159271. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3797550 .   \n\n\u2191 Nguyen, L.; Bellucci, E.; Nguyen, L.T. (2014). \"Electronic health records implementation: An evaluation of information system impact and contingency factors\". International Journal of Medical Informatics 83 (11): 779\u201396. doi:10.1016\/j.ijmedinf.2014.06.011. PMID 25085286.   \n\n\u2191 Moen, A.; Hackl, W.O.; Hofdijk, J. et al. (2013). \"eHealth in Europe - Status and challenges\". Yearbook of Medical Informatics 8: 59\u201363. PMID 23974549.   \n\n\u2191 Olsson, S.; Lymberis, A.; Whitehouse, D. (2004). \"European Commission activities in eHealth\". International Journal of Circumpolar Health 63 (4): 310-6. PMID 15709306.   \n\n\u2191 Iakovidis, I.; Purcarea, O. (2008). \"eHealth in Europe: From vision to reality\". Studies in Health Technology and Informatics 134: 163\u20138. PMID 18376043.   \n\n\u2191 14.0 14.1 Martins, C.; Azevedo, L.F.; Ribeiro, O. (2013). \"A population-based nationwide cross-sectional study on preventive health services utilization in Portugal - What services (and frequencies) are deemed necessary by patients?\". PLOS One 8 (11): e81256. doi:10.1371\/journal.pone.0081256. PMC PMC3836775. PMID 24278405. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3836775 .   \n\n\u2191 van Bokhoven, M.A.; Pleunis-van Empel, M.C.; Koch, H. et al. (2006). \"Why do patients want to have their blood tested? A qualitative study of patient expectations in general practice\". BMC Family Practice 7: 75. doi:10.1186\/1471-2296-7-75. PMC PMC1769380. PMID 17166263. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1769380 .   \n\n\u2191 van der Weijden, T.; van Bokhoven, M.A.; Dinant, G.J. et al. (2002). \"Understanding laboratory testing in diagnostic uncertainty: A qualitative study in general practice\". British Journal of General Practice 52 (485): 974\u201380. PMC PMC1314466. PMID 12528582. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1314466 .   \n\n\u2191 van Bokhoven, M.A.; Koch, H.; Dinant, G.J. et al. (2008). \"Exploring the black box of change in improving test-ordering routines\". Family Practice 25 (3): 139-45. doi:10.1093\/fampra\/cmn022. PMID 18535302.   \n\n\u2191 Verstappen, W.H.; ter Riet, G.; Dubois, W.I. et al. (2004). \"Variation in test ordering behaviour of GPs: Professional or context-related factors?\". Family Practice 21 (4): 387-95. doi:10.1093\/fampra\/cmh408. PMID 15249527.   \n\n\u2191 Main, C.; Moxham, T.; Wyatt, J.C. et al. (2010). \"Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: Systematic reviews of the effects and cost-effectiveness of systems\". Health Technology Assessment 14 (48): 1\u2013227. doi:10.3310\/hta14480. PMID 21034668.   \n\n\u2191 da Costa Pereira, A.; Giest, S.; Dumortier, J.; Artmann, J. (October 2010). \"eHealth Strategies - Country Brief: Portugal\" (PDF). European Commission. http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.476.4177&rep=rep1&type=pdf .   \n\n\u2191 U.S. Preventive Services Task Force (2014). \"The Guide to Clinical Preventive Services 2014\". https:\/\/www.overdrive.com\/media\/2062549\/clinical-preventive-services-2014 . Retrieved 08 October 2016 .   \n\n\u2191 22.0 22.1 Martins, C.; Azevedo, L.F.; Santos, C. (2014). \"Preventive health services implemented by family physicians in Portugal - A cross-sectional study based on two clinical scenarios\". BMJ Open 4 (5): e005162. doi:110.1136\/bmjopen-2014-005162. PMC PMC4039810. PMID 24861550. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4039810 .   \n\n\u2191 Administra\u00e7\u00e3o Regional de Sa\u00fade do Norte, IP (2010), Action plan 2010-2012 of the Western Oporto grouping of health centers   \n\n\u2191 da Sa\u00fade, M. (2007), National Plan for Prevention and Control of Oncological Diseases 2007\/2010   \n\n\u2191 Zaat, J.O.; van Eijk, J.T.; Bonte, H.A. (1992). \"Laboratory test form design influences test ordering by general practitioners in The Netherlands\". Medical Care 30 (3): 189\u201398. PMID 1538607.   \n\n\u2191 Barth, J.H.; Balen, A.H.; Jennings, A. (2001). \"Appropriate design of biochemistry request cards can promote the use of protocols and reduce unnecessary investigations\". Annals of Clinical Biochemistry 38 (Pt 6): 714\u20136. doi:10.1258\/0004563011900957. PMID 11732657.   \n\n\u2191 Bailey, J.; Jennings, A.; Parapia, L. (2005). \"Change of pathology request forms can reduce unwanted requests and tests\". Journal of Clinical Pathology 58 (8): 853\u20135. doi:10.1136\/jcp.2004.023101. PMC PMC1770878. PMID 16049288. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1770878 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Grammar edits were made to make the document more readable. The numerical data in the last quarter of Table 4 changed to two decimal places for no clear reason; however, it was retained as-is.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\">https:\/\/www.limswiki.org\/index.php\/Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 10 April 2017, at 22:41.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,054 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","45e8147ab685e78229d6e364a84469a5_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:The effect of a test ordering software intervention on the prescription of unnecessary laboratory tests - A randomized controlled trial<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: The way <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health record<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Computerized_physician_order_entry\" title=\"Computerized physician order entry\" target=\"_blank\" class=\"wiki-link\" data-key=\"f9e67e685f2b29f79e9b0991330f2b10\">laboratory test ordering system<\/a> software is designed may influence physicians\u2019 prescription. A randomized controlled trial was performed to measure the impact of a diagnostic and laboratory tests ordering system software modification.\n<\/p><p><b>Methods<\/b>: Participants were family physicians working and prescribing diagnostic and <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> tests.\n<\/p><p>The intervention group had modified software with basic shortcut menu changes, where some tests were withdrawn or added, and with the implementation of an evidence-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" target=\"_blank\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support system<\/a> based on United States Preventive Services Task Force (USPSTF) recommendations. This intervention group was compared with typically used software (control group).\n<\/p><p>The outcomes were the number of tests prescribed from those: withdrawn from the basic menu; added to the basic menu; marked with green dots (USPSTF\u2019s grade A and B); and marked with red dots (USPSTF\u2019s grade D).\n<\/p><p><b>Results<\/b>: Comparing the monthly average number of tests prescribed before and after the software modification, from those tests that were withdrawn from the basic menu, the control group prescribed 33.8 tests per 100 consultations before and 30.8 after (p\u2009=\u20090075); the intervention group prescribed 31.3 before and 13.9 after (p\u2009<\u20090001). Comparing the tests prescribed between both groups during the intervention, from those tests that were withdrawn from the basic menu, the intervention group prescribed a monthly average of 14.0 vs. 29.3 tests per 100 consultations in the control group (p\u2009<\u20090.001). From those tests that are USPSTF\u2019s grade A and B, the intervention group prescribed 66.8 vs. 74.1 tests per 100 consultations in the control group (p\u2009=\u20090.070). From those tests categorized as USPSTF grade D, the intervention group prescribed an average of 9.8 vs. 11.8 tests per 100 consultations in the control group (p\u2009=\u20090.003).\n<\/p><p><b>Conclusions<\/b>: Removing unnecessary tests from a quick shortcut menu of the diagnosis and laboratory tests ordering system had a significant impact and reduced unnecessary prescription of tests.\n<\/p><p>The fact that it was not possible to perform the randomization at the family physicians\u2019 level, but only on the computer servers is a limitation of our study. Future research should assess the impact of different test ordering systems during longer periods.\n<\/p><p><b>Trial registration<\/b>: <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.controlled-trials.com\/ISRCTN45427977\" target=\"_blank\">ISRCTN45427977<\/a>, May 1st 2014 (retrospectively registered).\n<\/p><p><b>Keywords<\/b>: Preventive health services, primary health care, evidence-based practice, decision support systems, clinical decision making, computer-assisted\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" target=\"_blank\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">Informatics<\/a> has undoubtedly changed the way societies live, socialize, learn, work, and deal with healthcare. We now live in a period of increasing concern about the excessive presence of medicine in our lives.<sup id=\"rdp-ebb-cite_ref-GlasziouTooMuch13_1-0\" class=\"reference\"><a href=\"#cite_note-GlasziouTooMuch13-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MoynihanPreventing12_2-0\" class=\"reference\"><a href=\"#cite_note-MoynihanPreventing12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GetzIsOpp03_3-0\" class=\"reference\"><a href=\"#cite_note-GetzIsOpp03-3\" rel=\"external_link\">[3]<\/a><\/sup> When inefficient software is combined with a non-evidence-based medical practice, there is the risk of patient harm, significant impact to quality of life, and damage to the healthcare system due to unnecessary costs.\n<\/p><p>The implementation of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a> (EHRs) has both potential benefits and drawbacks.<sup id=\"rdp-ebb-cite_ref-RaposoElect15_4-0\" class=\"reference\"><a href=\"#cite_note-RaposoElect15-4\" rel=\"external_link\">[4]<\/a><\/sup> Among the benefits, the prevention of medical errors and the promotion of patient safety has often been mentioned and confirmed in clinical practice.<sup id=\"rdp-ebb-cite_ref-RaposoElect15_4-1\" class=\"reference\"><a href=\"#cite_note-RaposoElect15-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Ben-AssuliElect15_5-0\" class=\"reference\"><a href=\"#cite_note-Ben-AssuliElect15-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Ben-AssuliAssess16_6-0\" class=\"reference\"><a href=\"#cite_note-Ben-AssuliAssess16-6\" rel=\"external_link\">[6]<\/a><\/sup> Despite the positive effects of EHR implementation in clinical practice, a range of barriers faced by physicians has been identified. These barriers may include technical and financial aspects, time, psychological, social, legal, and organizational changes to the process.<sup id=\"rdp-ebb-cite_ref-BoonstraBarriers10_7-0\" class=\"reference\"><a href=\"#cite_note-BoonstraBarriers10-7\" rel=\"external_link\">[7]<\/a><\/sup> After having removed the first barriers to EHR implementation, it is now time to implement continuing improvement and development of the available tools and to incorporate the scientific evidence obtained to this point.<sup id=\"rdp-ebb-cite_ref-RaposoElect15_4-2\" class=\"reference\"><a href=\"#cite_note-RaposoElect15-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RoukemaPaper06_8-0\" class=\"reference\"><a href=\"#cite_note-RoukemaPaper06-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BowmanImpact13_9-0\" class=\"reference\"><a href=\"#cite_note-BowmanImpact13-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NguyenElect14_10-0\" class=\"reference\"><a href=\"#cite_note-NguyenElect14-10\" rel=\"external_link\">[10]<\/a><\/sup>\n<\/p><p>To achieve better patient safety standards and improve healthcare system cost-effectiveness, there has been a worldwide effort to implement an integrated EHR system with diagnostic and <a href=\"https:\/\/www.limswiki.org\/index.php\/Computerized_physician_order_entry\" title=\"Computerized physician order entry\" target=\"_blank\" class=\"wiki-link\" data-key=\"f9e67e685f2b29f79e9b0991330f2b10\">laboratory test ordering communication systems<\/a>.<sup id=\"rdp-ebb-cite_ref-MoenEHealth13_11-0\" class=\"reference\"><a href=\"#cite_note-MoenEHealth13-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OlssonEuropean04_12-0\" class=\"reference\"><a href=\"#cite_note-OlssonEuropean04-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-IakovidisEHealth08_13-0\" class=\"reference\"><a href=\"#cite_note-IakovidisEHealth08-13\" rel=\"external_link\">[13]<\/a><\/sup> There have also been attempts to incorporate <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" target=\"_blank\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support systems<\/a> to further improve the quality of medicine. Prescribing diagnostic and laboratory tests is a key component of medical consultation. In the primary health care setting, tests are often ordered with preventive intentions and fulfillment of patient expectations.<sup id=\"rdp-ebb-cite_ref-MartinsAPop13_14-0\" class=\"reference\"><a href=\"#cite_note-MartinsAPop13-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-vanBokhovenWhy06_15-0\" class=\"reference\"><a href=\"#cite_note-vanBokhovenWhy06-15\" rel=\"external_link\">[15]<\/a><\/sup> There is also great uncertainty and variability among family physicians\u2019 ordering routines.<sup id=\"rdp-ebb-cite_ref-vanDerWeijdenUnder02_16-0\" class=\"reference\"><a href=\"#cite_note-vanDerWeijdenUnder02-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-vanBokhovenExploring08_17-0\" class=\"reference\"><a href=\"#cite_note-vanBokhovenExploring08-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VerstappenVariation04_18-0\" class=\"reference\"><a href=\"#cite_note-VerstappenVariation04-18\" rel=\"external_link\">[18]<\/a><\/sup> The effects of test ordering communication systems integrated with clinical decision support systems have been reported in various clinical practice settings. Main <i>et al.<\/i> have performed a systematic review of this topic and reported that integration of clinical decision support systems resulted in significant benefits to the prescribing process and practitioner performance outcomes in nearly two-thirds of the 24 studies that met the inclusion criteria.<sup id=\"rdp-ebb-cite_ref-MainComput10_19-0\" class=\"reference\"><a href=\"#cite_note-MainComput10-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/p><p>In Portugal, the use of EHR software with a diagnostic and laboratory test order communication system has been mandatory since September 2011. Most of the primary healthcare centers use software called <i>Sistema de Apoio ao M\u00e9dico<\/i> (Physician\u2019s Support System [SAM]). In the module used to order diagnostic and laboratory tests, physicians access a searchable test menu by two possible strategies: 1) typing the test name in a search box or 2) browsing by a shortcut menu structure (Fig. 1). Different menus are available for most areas of medicine, including basic, allergology, andrology, cardiovascular, central nervous system, digestive, dosing, endocrinology, gynecology, hematology, infectious diseases, nephrology, obstetrics, oncology, otorhinolaryngology, osteoarticular, preoperative, respiratory, rheumatology, and urology. Under each menu there is a set of specific lab tests. Physicians can choose one or more tests by double-clicking each test or can choose the entire set by double-clicking on the shortcut menu\u2019s title. For example, the basic menu is composed of uric acid, total cholesterol, creatinine, gamma-glutamyl transferase, glucose, hemogram, serum protein electrophoresis, aspartate aminotransferase, urine type 2, sedimentation rate, electrocardiogram, and lung x-ray tests.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Martins_BMCMedInfoDecMak2017_17-1.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5f71295a3ad1a4e340686ff98c36eed1\"><img alt=\"Fig1 Martins BMCMedInfoDecMak2017 17-1.gif\" src=\"https:\/\/www.limswiki.org\/images\/8\/86\/Fig1_Martins_BMCMedInfoDecMak2017_17-1.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Usual ordering communication system: the basic shortcut menu<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>If a physician would like to choose only hemogram and glucose, they must double-click on each test. However, if they would like the entire basic set of tests, they must double-click on the basic menu title. Our research team suspected that this basic shortcut menu is often selected during routine consultations in which patients ask for routine check-ups without specific reason. As we have shown in a previous study, there is a high prevalence of Portuguese adults (99.2%) that believe they should have routine blood and urine tests annually.<sup id=\"rdp-ebb-cite_ref-MartinsAPop13_14-1\" class=\"reference\"><a href=\"#cite_note-MartinsAPop13-14\" rel=\"external_link\">[14]<\/a><\/sup> This statistic demonstrates the importance of examining the effectiveness and efficiency of this basic sub-menu.\n<\/p><p>Through a randomized controlled trial, the primary aim of the present study was to compare the effects of modifying the EHR ordering communication system (modified SAM) by changing the basic shortcut menu and adding a clinical decision support system based on the integration of the United States Preventive Services Task Force (USPSTF) recommendations. After the last primary healthcare reform in Portugal, primary healthcare centers have been divided into healthcare center groups.<sup id=\"rdp-ebb-cite_ref-daCostaPereiraEHealth10_20-0\" class=\"reference\"><a href=\"#cite_note-daCostaPereiraEHealth10-20\" rel=\"external_link\">[20]<\/a><\/sup> In a healthcare center group, the informatics network is linked through servers that may serve more than one healthcare center. Creating a modified version of the SAM software requires it to be installed at a server level, which determines that all physicians at all healthcare centers served by that server will receive the same version of the software. For this reason, it was not possible to randomize the study at the physician level. Rather, we had to randomize the servers at the healthcare center group level.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Trial_design\">Trial design<\/span><\/h3>\n<p>All servers of the Western Oporto grouping of health centers counted for randomization, except that serving the center where study authors worked (to avoid possible bias). The remaining nine servers were randomized into two groups: 1) five servers were randomly allocated to the intervention group and 2) four servers to the control group.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Participants\">Participants<\/span><\/h3>\n<p>All family physicians working and prescribing diagnostic and laboratory tests in the Western Oporto group of health centers (except in those where the authors worked) participated in this study.\n<\/p><p>Data of the diagnostic and laboratory tests prescriptions were centrally collected by informatics staff belonging to the Ministry of Health and sent to the research team without patients\u2019 or physicians\u2019 identifications.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Interventions\">Interventions<\/span><\/h3>\n<p>The control group continued to use the standard version of the EHR software (SAM) as presented in Fig. 1. The intervention group used a modified version of the software (SAM modified) installed in each server (Fig. 2). Software modifications consisted of two principal changes: \n<\/p><p>1. Basic shortcut menu changes: The composition of the basic menu set of diagnostic laboratory tests was changed. Some tests were removed, including uric acid, serum protein electrophoresis, sedimentation rate, electrocardiogram, and lung x-ray. Other tests were added, including HDL cholesterol, fecal occult blood test, triglycerides, Pap smear, and mammography. Although some tests were removed from the basic menu, physicians were still able to request them by typing their names in the search for tests box.\n<\/p><p>2. Addition of an evidence-based decision support: For the tests listed in Table 1, we added traffic light-based colored dots according to the USPSTF recommendations and an additional information box containing the summary of the USPSTF recommendation and a link to the integral recommendation at the USPSTF website (Fig. 2).<sup id=\"rdp-ebb-cite_ref-USPSTFTheGuide14_21-0\" class=\"reference\"><a href=\"#cite_note-USPSTFTheGuide14-21\" rel=\"external_link\">[21]<\/a><\/sup>\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Martins_BMCMedInfoDecMak2017_17-1.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"66d4aa892b40ec1b474d25abf6e1db51\"><img alt=\"Fig2 Martins BMCMedInfoDecMak2017 17-1.gif\" src=\"https:\/\/www.limswiki.org\/images\/b\/bb\/Fig2_Martins_BMCMedInfoDecMak2017_17-1.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Modified ordering communication system: the basic shortcut menu. Red numbers: 1 - Traffic lights colored dots according to United States Preventive Services Task Force recommendations grades. 2 - Text box with the summary of the recommendation for each selected test. 3 - Link to the original recommendation at the USPSTF\u2019s website. 4 - Search for tests box, where any test, including those removed from the basic menu, can be searched and requested by typing the test\u2019s name. 5 - Legend of the colored dots<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><b>Table 1.<\/b> United States Preventives Services Task Force (USPSTF) recommendations (March, 2012)<br \/><sup>a<\/sup> USPSTF grades A and B: the USPSTF recommends the service, marked with green dots. Grade D: the USPSTF recommends against the service, marked with red dots. Grade I: the USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of the service, marked with yellow dots.<br \/>\n<p><sup>b<\/sup> <i>DXA<\/i> Dual-energy X-ray absorptiometry\n<\/p>\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Colored dots\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Grade<sup>a<\/sup>\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Test\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Summary of the recommendations\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pelvic ultrasound\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not recommended as routine screening for ovarian cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer antigen 19-9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not recommended as routine screening for pancreatic cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Rest electrocardiography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not recommended in asymptomatic adults at low risk for coronary heart disease events\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Exercise electrocardiography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not recommended in asymptomatic adults at low risk for coronary heart disease events\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Carotid artery ultrasound\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not recommended as screening for asymptomatic carotid artery stenosis in the general adult population\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Spirometry\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not recommended as screening adults for chronic obstructive pulmonary disease\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hepatitis B surface antigen\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not recommended as routine screening\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hepatitis C antibodies\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not recommended as routine screening\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yellow\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Triglycerides\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">There is currently insufficient evidence of the benefit of including triglycerides as a part of the initial tests used to screen routinely for dyslipidemia.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yellow\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Prostate-specific antigen\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The evidence is insufficient to recommend for or against routine screening for prostate cancer in men younger than 75.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yellow\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lung computed tomography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The evidence is insufficient to recommend for or against routine screening for lung cancer.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yellow\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lung x-ray\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The evidence is insufficient to recommend for or against routine screening for lung cancer.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yellow\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Thyroid-stimulating hormone\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The evidence is insufficient to recommend for or against routine screening.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">B\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Glucose\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Screening for type 2 diabetes recommended in asymptomatic adults with sustained blood pressure (either treated or untreated) greater than 135\/80 mm Hg\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\/B\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total cholesterol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Recommended every 5 years, for men aged 35 and older and women 45 and older if they are at increased risk for coronary heart disease, at younger ages if they are at increased risk for coronary heart disease.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">B\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mammography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Biennial screening recommended for women aged 50 to 74 years\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cervicovaginal cytology\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Every 3 years screening recommended for women who have cervix, 21 to 65 years old\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fecal occult blood test\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Recommended annually as a possible method of screening for colorectal cancer, 50 to 75 years\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Colonoscopy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Recommended every 10 years as a possible method of screening for colorectal cancer, 50 to 75 years\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Flexible sigmoidoscopy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Recommended every 5 years as a possible method of screening for colorectal cancer, 50 to 75 years\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\/B\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HDL cholesterol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Recommended every 5 years, for men aged 35 and older and women 45 and older if they are at increased risk for coronary heart disease, at younger ages if they are at increased risk for coronary heart disease.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">B\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DXA<sup>b<\/sup> of the hip and lumbar spine\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Screening recommended for osteoporosis in women aged 65 years or older and in younger women whose fracture risk is equal to or greater than that of a 65-year-old\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Venereal Disease Research Laboratory\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Recommended for persons at increased risk for syphilis infection\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The selection of the specific tests to be removed from or added to the basic shortcut menu was made after discussion and debate with eight family physicians selected by proximity and convenience. The main rational was to remove tests that are not recommended to be used as routine screening tools and to substitute them with others that may be recommended for certain risk groups. However, creating a change that was not too disruptive to the current family physicians\u2019 practices was a concern, and that is why tests such as creatinine, gamma-glutamyl transferase, hemogram, aspartate aminotransferase, and urine type 2 were left in the basic shortcut menu.\n<\/p><p>EHR software modifications were implemented on May 30\u201331, 2012 in all of the intervention groups\u2019 servers. Prospective monthly monitoring and data collection occurred until January 31, 2013. To allow a pre-post analysis in both groups, a retrospective monthly data collection of both control and intervention groups was also performed between December 1, 2011 and May 31, 2012.\n<\/p><p>The collected data included monthly parameters, including numbers of prescribing family physicians, face-to-face consultations made, and number of each prescribed diagnostic and laboratory test.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Independent_variables\">Independent variables<\/span><\/h3>\n<p>Group (control and intervention) and time (before and after the modifications) were the independent variables.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Dependent_variables\">Dependent variables<\/span><\/h3>\n<p>Primary outcomes were chosen to assess the impact of our intervention on the number of diagnostic and laboratory tests prescribed by physicians using four different perspectives: 1) impact on the number of the prescriptions of diagnostic and laboratory tests that were withdrawn from the basic menu; 2) impact on the number of the prescriptions of diagnostic and laboratory tests that were added to the basic menu; 3) impact on the number of the prescriptions of diagnostic and laboratory tests that were marked with green dots (USPSTF recommendation grades A and B); and 4) impact on the number of prescriptions of diagnostic and laboratory tests that were marked with red dots (USPSTF recommendation grade D). Diagnostic and laboratory tests were the dependent variables.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sample_size\">Sample size<\/span><\/h3>\n<p>Given administrative and technical barriers related to the permission process for the introduction of EHR software modifications, our sample size was obtained by convenience. We decided to run this trial at the Western Oporto grouping of healthcare centers, which includes a total of 15 primary care healthcare centers and an informatics network of 10 servers. One of the servers, which served the healthcare center in which some of the study authors worked, was excluded from the study.\n<\/p><p>The power calculation was done for the mean number of monthly prescribed tests per 100 consultations comparison between control and intervention groups. A significance level of 0.05, a power of 0.8, and a standard deviation of two tests prescribed per 100 consultations per month was considered. A mean difference of 3.3 tests prescribed per 100 consultations per month between control and intervention groups was used.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Randomization\">Randomization<\/span><\/h3>\n<p>The remaining nine servers were sequentially numerated and randomly assigned to the intervention and control group by an investigator blinded to the server identification. The allocation sequence was computer generated resulting in five servers (seven healthcare centers, 58 family physicians) allocated to the intervention group, and four servers (seven healthcare centers, 59 family physicians) allocated to the control group.\n<\/p><p>To guarantee allocation anonymity, family physicians at each healthcare center only received information about this trial implementation after randomization has been performed. Given the nature of this trial, no consent at physician level was obtained. Consent was obtained from the Northern Regional Health Administration and the Executive Council of the Western Oporto Group of Health Centers.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Statistical_methods\">Statistical methods<\/span><\/h3>\n<p>To examine whether the software modification changed test prescription trends we performed an interrupted time series analysis with an autoregressive integrated moving average (ARIMA) model, using the monthly number of tests prescribed per 100 consultations and the intervention (the software modification) as a dichotomous variable (before and after intervention). In the control group, we also performed an ARIMA model analysis using the monthly number of tests prescribed per 100 consultations and the same dichotomous variable (before and after intervention), although there was no intervention in this group. The before and after intervention comparison of the monthly average number of diagnostic and lab tests prescribed per 100 consultations between control and intervention groups was made using an independent sample t test. The Bonferroni correction was used for adjusting for multiple testing (several diagnostic and laboratory tests).<sup id=\"rdp-ebb-cite_ref-MartinsPreventive14_22-0\" class=\"reference\"><a href=\"#cite_note-MartinsPreventive14-22\" rel=\"external_link\">[22]<\/a><\/sup> A significance level of 0.05 was used. As size effect measure, Cohen\u2019s d was calculated and a d of \u22650.8 was considered a large effect.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Ethical_considerations\">Ethical considerations<\/span><\/h3>\n<p>This study was approved by the Northern Regional Health Administration Medical Ethics Committee with several considerations: \u201c1) Given that the study will collect only anonymous data there is no need for the obtainment of informed consent by physicians; 2) The study is of great relevance and with an expected practical interest of the results; and 3) The methodology used safeguards the rights of participants.\u201d\n<\/p><p>In accordance with the Medical Ethics Committee, before the implementation of the study (but after the randomized allocation), a letter was sent to all Western Oporto Group of healthcare centers informing them of and explaining the study aims and methodology.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>The number of servers enrolled was constant without loss (Fig. 3). The number of family physicians after the intervention ranged from 58 to 64 in the intervention group and from 55 to 70 in the control group. Recruitment was performed in April 2012, and the trial began on June 1, 2012, with a follow-up period until January 31, 2013 (eight month follow-up). Baseline data are presented in Table 2.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Martins_BMCMedInfoDecMak2017_17-1.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"dccd54d57f99a95fbf14ab0d6cba4ab2\"><img alt=\"Fig3 Martins BMCMedInfoDecMak2017 17-1.gif\" src=\"https:\/\/www.limswiki.org\/images\/d\/dc\/Fig3_Martins_BMCMedInfoDecMak2017_17-1.gif\" width=\"358\" height=\"302\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Flowchart summary of the trial<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><b>Table 2.<\/b> Baseline data ranges for each group based on the last six months before the electronic health record (EHR) software modification\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Usual software (control group) min - max\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Modified software (intervention group) min - max\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Family physicians\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">57\u201361\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">55\u201363\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total of patients covered by health centers\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">110,781\u2013110,783\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">79,847\u201379,850\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Face-to-face consultations\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10,725\u201313,493\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10,047\u201312,598\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For both groups, Table 3 compares the monthly average number of diagnostic and laboratory tests prescribed for each 100 consultations before and after EHR software modification, under a time series analysis perspective (ARIMA model). Within the five tests that were withdrawn from the basic menu, we observe a statistically significant reduction in the prescription of four tests (uric acid, serum protein electrophoresis, sedimentation rate, and lung x-ray tests) in the intervention group after software modification. Within the five tests that were added to the basic menu, we observed a significant increase in prescription of the fecal occult blood test in the intervention group after software modification. Within the tests that were marked with green dots (USPSTF recommendation grades A and B) there were no significant increases in either the control or intervention groups. Within the tests that were marked with red dots (USPSTF recommendation grade D), there was a significant reduction only in the prescription of cancer antigen 19-9, but the amount of this test that was prescribed was so low that this variation may be justified by other factors and not a result of software modification.\n<\/p><p><br \/>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><b>Table 3.<\/b> Pre-post comparison of the monthly number of diagnostic and lab tests prescribed per 100 consultations (time series analysis, ARIMA model)<br \/>\n<p><sup>a<\/sup> Tests that were also marked with red dots<br \/>\n<sup>b<\/sup> Tests that were also marked with green dots<br \/>\n<sup>\u03b1<\/sup> DXA: Dual-energy X-ray absorptiometry<br \/>\n<sup>\u03b2<\/sup> VDRL: Venereal Disease Research Laboratory\n<\/p>\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\" rowspan=\"2\" colspan=\"1\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"3\">Control group (always exposed to usual EHR software)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"3\">Intervention group\n<\/th><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Before\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">After\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\"><i>p<\/i>\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Before software modification\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">After software modification\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\"><i>p<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><i>Diagnostic and lab tests that were withdrawn from the basic menu<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Uric acid\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.094\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Serum protein electrophoresis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sedimentation rate\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.259\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Rest electrocardiography<sup>a<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.189\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.072\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lung x-ray\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.351\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.002\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>All tests withdrawn<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.075\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><i>Diagnostic and laboratory tests that were added to the basic menu<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HDL cholesterol<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.357\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.566\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Triglycerides\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.349\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.372\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fecal occult blood test<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.236\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.017\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cervicovaginal cytology<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.779\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.905\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mammography<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.332\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.631\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>All tests added<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">53.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">49.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.334\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">45.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">44.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.909\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><i>Diagnostic and laboratory tests that were marked with green dots (USPSTF grade A and B)<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Glucose\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">21.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.402\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.199\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total cholesterol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.345\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.264\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Colonoscopy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.325\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.940\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Flexible sigmoidoscopy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.031\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.115\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DXA<sup>\u03b1<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.035\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.562\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">VDRL<sup>\u03b2<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.234\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.003\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>All green marked<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">76.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.331\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">70.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">67.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.560\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><i>Diagnostic and laboratory tests that were marked with red dots (USPSTF grade D)<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pelvic ultrasound\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.033\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.815\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer antigen 19-9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.911\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.013\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Exercise electrocardiography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.308\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.113\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Carotid artery ultrasound\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.484\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.072\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Spirometry\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.959\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.010\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hepatitis B surface antigen\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.627\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.147\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hepatitis C antibodies\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.379\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.898\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>All red marked<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.341\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.102\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Table 4 presents a direct comparison between the control and intervention groups before and after software modifications (excluding the first month following modification that was considered as a washout period). Regarding the set of tests that were withdrawn from the basic menu, we observe that there were no significant differences between either group before software modification, but that there was a significantly lower prescription rate in the intervention group for the following tests after software modification: 1) uric acid; 2) serum protein electrophoresis; and 3) sedimentation rate. When we consider all tests belonging to the set withdrawn from the basic menu, we have verified that after software modification the intervention group prescribed less than half of those tests when compared with the control group (14.01 tests per 100 consultations versus 29.29 tests per 100 consultations, p\u2009<\u20090.001). Regarding the five tests that were added to the basic menu, we observed no significant differences after software modification between the two groups. Considering the tests that were marked with green dots (USPSTF recommendation grades A and B) we observed no significantly higher prescription rates in the intervention group after the software modification compared to the control group. For the tests that were marked with red dots (USPSTF recommendation grade D), we observed significant lower prescription rates in five tests (cancer antigen 19-9, exercise electrocardiography, carotid artery ultrasound, hepatitis B surface antigen, and hepatitis C antibodies) in the intervention group after software modification.\n<\/p><p><br \/>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"9\"><b>Table 4.<\/b> Comparison of the monthly number of diagnostic and lab tests prescribed per 100 consultations between control and intervention groups, before and after EHR software modification (independent sample t test with Bonferroni correction)\n<p><sup>a<\/sup> Tests that were also marked with red dots<br \/>\n<sup>b<\/sup> Tests that were also marked with green dots<br \/>\n<sup>\u03b1<\/sup> DXA: Dual-energy X-ray absorptiometry<br \/>\n<sup>\u03b2<\/sup> VDRL: Venereal Disease Research Laboratory\n<\/p>\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\" rowspan=\"2\" colspan=\"1\">\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"4\">Before (always exposed to usual EHR software)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"4\">After software modification (first month, washout period, excluded)\n<\/th><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Control\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Intervention\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\"><i>p<\/i>\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Cohen's d\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Control\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Intervention\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\"><i>p<\/i>\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Cohen's d\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><i>Diagnostic and lab tests that were withdrawn from the basic menu<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Uric acid\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.632\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.138\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Serum protein electrophoresis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.605\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-1.039\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.384\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sedimentation rate\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.315\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.207\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.818\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Rest electrocardiography<sup>a<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.465\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.073\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.070\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.544\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lung x-ray\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.985\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.802\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.441\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>All tests withdrawn<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.292\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.641\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29.29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.174\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><i>Diagnostic and laboratory tests that were added to the basic menu<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HDL cholesterol<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.640\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.958\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.702\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Triglycerides\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.830\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.862\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.640\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.874\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fecal occult blood test<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.015\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.207\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.425\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.003\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cervicovaginal cytology<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.468\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.361\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mammography<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.855\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.851\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.157\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>All tests added<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">53.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">45.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.070\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.172\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">48.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">44.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.158\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.805\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><i>Diagnostic and laboratory tests that were marked with green dots (USPSTF grade A and B)<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Glucose\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.864\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.914\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.228\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.246\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total cholesterol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.930\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.885\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.450\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.040\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Colonoscopy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.042\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.943\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.006\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.345\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Flexible sigmoidoscopy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.002\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.186\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-1.326\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DXA<sup>\u03b1<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.036\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.010\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.414\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.063\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">VDRL<sup>\u03b2<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.552\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.077\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.102\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.483\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>All green marked<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">70.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.070\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.170\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">74.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">66.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.070\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.064\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><i>Diagnostic and laboratory tests that were marked with red dots (USPSTF grade D)<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pelvic ultrasound\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.084\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.774\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.305\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer antigen 19-9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.18\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.616\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.093\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.513\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Exercise electrocardiography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.52\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.48\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.416\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.56\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.38\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.028\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.900\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Carotid artery ultrasound\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.750\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.021\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.978\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Spirometry\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.47\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.021\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.307\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.45\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.52\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.812\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.902\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hepatitis B surface antigen\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.60\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.084\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.756\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.59\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.028\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.925\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hepatitis C antibodies\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.91\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.021\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.207\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.18\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.91\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.014\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.045\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>All red marked<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.021\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.583\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.003\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.941\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>Our results show that removing unnecessary tests from a quick shortcut menu of diagnostic and laboratory tests available on the EHR test ordering system might have significantly impacted ordering habits and reduced unnecessary test prescriptions. This result emphasized the importance of careful attention and scientific rigor when building shortcut menus in ordering systems. This impact was significant, and it could be observed from two perspectives of analysis: 1) when we compared the results for the intervention group before and after the modification of the software and 2) when we compared the intervention group with the control group.\n<\/p><p>Furthermore, our results demonstrated that the introduction of an evidence-based decision support tool had no significant impact on the test prescription profiles. However, it should be noted that as a passive instrument, the decision for support addition did not interfere with the normal flow of the ordering system. The doctors had only visual contact with the colored recommendations and optional access to the additional information from the USPSTF recommendations. The absence of impact of this support decision tool may also be related to a high level of agreement between the way Portuguese family doctors perform preventive health services and the USPSTF recommendations as shown in a previous national cross-sectional study.<sup id=\"rdp-ebb-cite_ref-MartinsPreventive14_22-1\" class=\"reference\"><a href=\"#cite_note-MartinsPreventive14-22\" rel=\"external_link\">[22]<\/a><\/sup>\n<\/p><p>Limitations of our study include the sampling method and sample size. The moderate size of the sample might have limited our conclusions regarding the less frequently prescribed tests, but this is unlikely to have affected the main conclusion of our study.\n<\/p><p>The fact that it was not possible to perform randomization at the family physician level, but only at that of the computer servers, also constitutes a limitation and a possible biasing factor, since the number of servers in the Western Oporto Grouping of healthcare centers was small. In the Western Oporto Grouping of healthcare centers we had nine servers. Since our intervention was a software modification, we had to randomize at the level of the electronic server. One hypothesis would be used to consider each server as a cluster, but the problem is that some servers serve only one health unit, not being a real cluster, while others serve two or three healthcare units. The ideal would be to consider each healthcare unit as a cluster of family physicians. For that, however, we had to have access to the number of tests prescribed by each family physician, which we did not. The Ministry of Health only allowed us access to the tests prescribed by each healthcare unit. This is the reason why it is not possible to consider this study as a cluster randomized controlled trial. Without the number of tests prescribed by each family physician, we were not able to calculate the intracluster correlation coefficient.\n<\/p><p>Another limitation of our study is that the time available for its completion was relatively short. An extended study period could have allowed us to draw more extensive conclusions (for example, regarding the tests that are prescribed less frequently). A prolonged study period would also have diluted possible seasonal effects that might exist in seeking medical consultations and also minimized the impact of factors inherent in the economic situation of Portugal during the study period. In the context of the economic crisis, there was an increase in user fees paid by patients for medical consultations and laboratory tests in addition to Ministry of Health publications of clinical guidelines for prescriptions of some diagnostic and laboratory tests. These facts may have contributed to the gradual decrease in test prescriptions that we have observed during the study period both in the control and intervention groups. The extension of the study might also have diluted this effect and could have allowed a clearer reading of the impact of software modifications. However, the socio-economic status of intervention and control medical centers was identical. All healthcare centers involved in the study are located in the urban area of Oporto in a geographic area of approximately 22Km<sup>2<\/sup>.<sup id=\"rdp-ebb-cite_ref-AdminAction10_23-0\" class=\"reference\"><a href=\"#cite_note-AdminAction10-23\" rel=\"external_link\">[23]<\/a><\/sup> All study physicians were exposed to the same Ministry of Health guidelines and general education materials.\n<\/p><p>These limitations were mainly due to technical issues related to the fact that our research could not disrupt the normal functioning of resources and services of the Ministry of Health. However, we consider it unlikely that any of these limitations limited the main conclusion of this study: the considerable impact of unnecessary test removal in the shortcut menu of the ordering system.\n<\/p><p>Considering the potential generalizability of our results to other groups of healthcare centers of the Portuguese National Health Service, we believe that there is a high probability that the same effect will be found with similar software intervention. We believe that this is the case for two reasons: 1) the size of the obtained effect was evident and 2) the main features of the ordering system of other groups of healthcare centers are similar.\n<\/p><p>For this study, we used the recommendations of the USPSTF as our standard guidelines because it bases its recommendations on freely accessible, evidence-based systematic reviews with recognized methodological quality that covers a considerable range of topics. Portugal is in a phase of transition regarding clinical guidelines and recommendations. The Ministry of Health has recently published guidelines covering some topics related with test prescriptions. However, many of these guidelines are still in the public discussion phase and are not based on systematic reviews but rather on experts' opinion. For these reasons we chose the USPSTF recommendations. Exceptions to this are the recommendations of the Portuguese Ministry of Health for three tests: 1) breast cancer screening by mammography every two years for women 50\u201369 years old; 2) colorectal cancer screening by a fecal occult blood test every one to two years for adults 50\u201374 years old; and 3) cervical cancer screening with a cervicovaginal cytology for women between 25 and 60 years every three years after two annual normal tests.<sup id=\"rdp-ebb-cite_ref-daSa.C3.BAdeNational10_24-0\" class=\"reference\"><a href=\"#cite_note-daSa.C3.BAdeNational10-24\" rel=\"external_link\">[24]<\/a><\/sup> These recommendations are in line with those from USPSTF but could have influenced our results.\n<\/p><p>When designing this trial, the researchers also considered the possibility of causing some negative effects as a result of the intervention. There could be two main type of negative effects: 1) on physicians because they could not find the tests in the basic shortcut menu as they previously had, and it would consume more consultation time to type the name in the search box to order one of the removed tests and 2) on patients because they could be deprived of some necessary tests. After debating this issue with some family physicians, these potential negative effects were considered negligible, mainly because there was no interference with physicians' freedom to prescribe any medical test, and the use of the search box as an alternative to the shortcut menu was not considered a major obstacle.\n<\/p><p>The reduction of unnecessary test prescriptions that we verified was in line with other studies in which changes in laboratory request forms also resulted in the reduction of unnecessary tests requests.<sup id=\"rdp-ebb-cite_ref-ZaatLab92_25-0\" class=\"reference\"><a href=\"#cite_note-ZaatLab92-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BarthApprop01_26-0\" class=\"reference\"><a href=\"#cite_note-BarthApprop01-26\" rel=\"external_link\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BaileyChange05_27-0\" class=\"reference\"><a href=\"#cite_note-BaileyChange05-27\" rel=\"external_link\">[27]<\/a><\/sup> However, there are some practical implications that may result from this study. For example, the Portuguese Health Authorities could use these results to improve test ordering software and to reduce the prescription of unnecessary tests at a national level. This study also reinforces the importance of adequately testing medical related software tools, and we hope that in the future this will be the rule for other medical software tools.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Our results might have a significant impact on improving the design of shortcut menus of diagnostic and laboratory test ordering systems either in the Portuguese National Health Service or in other countries\u2019 health systems. These improvements can help reduce the prescription of unnecessary tests, leading to the reduction of negative patient effects and to the reduction of unnecessary costs. These study results demonstrate the importance of testing and evaluating various aspects of medical informatics programs to improve efficiency and contribute to improved clinical practice and clinical outcomes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>ARIMA<\/b>: Autoregressive integrated moving average\n<\/p><p><b>EHR<\/b>: Electronic health records\n<\/p><p><b>SAM<\/b>: Sistema de Apoio ao M\u00e9dico (Physician\u2019s Support System)\n<\/p><p><b>USPSTF<\/b>: United States Preventive Services Task Force\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Declarations\">Declarations<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h3>\n<p>Not applicable.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>We have received financial support of 15,000 Eur from the Astrazeneca Portugal Foundation\u2019s 2010 program for support of research. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The co-author Andreia Teixeira thanks the financial support by FCT (SFRH\/BPD\/86383\/2012).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Availability_of_data_and_materials\">Availability of data and materials<\/span><\/h3>\n<p>The data are not available because the ethics committee only provided authorization for the research team to perform the raw data analysis.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authors.27_contributions\">Authors' contributions<\/span><\/h3>\n<p>CMSM had the original idea of the study, designed the trial and made the quality control during the data collection, drafted and revised the paper. LFRA contributted to the trial design and revised the paper. ASCT and CMNCS planned, supervised and has performed the statistical data analysis, contributed to the paper draft and revised the paper. LMBS, PAAPS, MLGDC, AMRCP and AAOPH revised the paper. All the authors read and approved the final version of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Consent_for_publication\">Consent for publication<\/span><\/h3>\n<p>Not applicable.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Ethics_approval_and_consent_to_participate\">Ethics approval and consent to participate<\/span><\/h3>\n<p>This study was approved by the Northern Regional Health Administration Medical Ethics Committee that considered: a) Given that the study will collect only anonymous data there is no need for the obtainment of informed consent by physicians; b) The study is of great relevance and with an expected practical interest of the results; and c) the methodology used safeguards the rights of participants.\n<\/p><p>In accordance with the Medical Ethics Committee, before the implementation of the study (but after the randomized allocation), a letter was sent to all Western Oporto Group of health centers informing and explaining the aims and methodology of the study.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-GlasziouTooMuch13-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GlasziouTooMuch13_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Glasziou, P.; Moynihan, R.; Richards, T. et al. (2013). \"Too much medicine; too little care\". <i>BMJ<\/i> <b>347<\/b>: f4247. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.f4247\" target=\"_blank\">10.1136\/bmj.f4247<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23820022\" target=\"_blank\">23820022<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Too+much+medicine%3B+too+little+care&rft.jtitle=BMJ&rft.aulast=Glasziou%2C+P.%3B+Moynihan%2C+R.%3B+Richards%2C+T.+et+al.&rft.au=Glasziou%2C+P.%3B+Moynihan%2C+R.%3B+Richards%2C+T.+et+al.&rft.date=2013&rft.volume=347&rft.pages=f4247&rft_id=info:doi\/10.1136%2Fbmj.f4247&rft_id=info:pmid\/23820022&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoynihanPreventing12-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoynihanPreventing12_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moynihan, R.; Doust, J.; Henry, D. (2012). \"Preventing overdiagnosis: How to stop harming the healthy\". <i>BMJ<\/i> <b>344<\/b>: e3502. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.e3502\" target=\"_blank\">10.1136\/bmj.e3502<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22645185\" target=\"_blank\">22645185<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Preventing+overdiagnosis%3A+How+to+stop+harming+the+healthy&rft.jtitle=BMJ&rft.aulast=Moynihan%2C+R.%3B+Doust%2C+J.%3B+Henry%2C+D.&rft.au=Moynihan%2C+R.%3B+Doust%2C+J.%3B+Henry%2C+D.&rft.date=2012&rft.volume=344&rft.pages=e3502&rft_id=info:doi\/10.1136%2Fbmj.e3502&rft_id=info:pmid\/22645185&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GetzIsOpp03-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GetzIsOpp03_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Getz, L.; Sigurdsson, J.A.; Hetlevik, I. (2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC188390\" target=\"_blank\">\"Is opportunistic disease prevention in the consultation ethically justifiable?\"<\/a>. <i>BMJ<\/i> <b>327<\/b> (7413): 498\u2013500. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.327.7413.498\" target=\"_blank\">10.1136\/bmj.327.7413.498<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC188390\/\" target=\"_blank\">PMC188390<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12946974\" target=\"_blank\">12946974<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC188390\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC188390<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Is+opportunistic+disease+prevention+in+the+consultation+ethically+justifiable%3F&rft.jtitle=BMJ&rft.aulast=Getz%2C+L.%3B+Sigurdsson%2C+J.A.%3B+Hetlevik%2C+I.&rft.au=Getz%2C+L.%3B+Sigurdsson%2C+J.A.%3B+Hetlevik%2C+I.&rft.date=2003&rft.volume=327&rft.issue=7413&rft.pages=498%E2%80%93500&rft_id=info:doi\/10.1136%2Fbmj.327.7413.498&rft_id=info:pmc\/PMC188390&rft_id=info:pmid\/12946974&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC188390&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RaposoElect15-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RaposoElect15_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-RaposoElect15_4-1\" rel=\"external_link\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-RaposoElect15_4-2\" rel=\"external_link\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Raposo, V.L. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4677576\" target=\"_blank\">\"Electronic health records: Is it a risk worth taking in healthcare delivery?\"<\/a>. <i>GMS Health Technology Assessment<\/i> <b>11<\/b>: Doc02. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3205%2Fhta000123\" target=\"_blank\">10.3205\/hta000123<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4677576\/\" target=\"_blank\">PMC4677576<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26693253\" target=\"_blank\">26693253<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4677576\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4677576<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+health+records%3A+Is+it+a+risk+worth+taking+in+healthcare+delivery%3F&rft.jtitle=GMS+Health+Technology+Assessment&rft.aulast=Raposo%2C+V.L.&rft.au=Raposo%2C+V.L.&rft.date=2015&rft.volume=11&rft.pages=Doc02&rft_id=info:doi\/10.3205%2Fhta000123&rft_id=info:pmc\/PMC4677576&rft_id=info:pmid\/26693253&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4677576&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Ben-AssuliElect15-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Ben-AssuliElect15_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ben-Assuli, O. (2015). \"Electronic health records, adoption, quality of care, legal and privacy issues and their implementation in emergency departments\". <i>Health Policy<\/i> <b>119<\/b> (3): 287\u201397. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.healthpol.2014.11.014\" target=\"_blank\">10.1016\/j.healthpol.2014.11.014<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25483873\" target=\"_blank\">25483873<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+health+records%2C+adoption%2C+quality+of+care%2C+legal+and+privacy+issues+and+their+implementation+in+emergency+departments&rft.jtitle=Health+Policy&rft.aulast=Ben-Assuli%2C+O.&rft.au=Ben-Assuli%2C+O.&rft.date=2015&rft.volume=119&rft.issue=3&rft.pages=287%E2%80%9397&rft_id=info:doi\/10.1016%2Fj.healthpol.2014.11.014&rft_id=info:pmid\/25483873&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Ben-AssuliAssess16-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Ben-AssuliAssess16_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ben-Assuli, O.; Leshno, M. (2016). \"Assessing electronic health record systems in emergency departments: Using a decision analytic Bayesian model\". <i>Health Informatics Journal<\/i> <b>22<\/b> (3): 712\u201329. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1460458215584203\" target=\"_blank\">10.1177\/1460458215584203<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26033468\" target=\"_blank\">26033468<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Assessing+electronic+health+record+systems+in+emergency+departments%3A+Using+a+decision+analytic+Bayesian+model&rft.jtitle=Health+Informatics+Journal&rft.aulast=Ben-Assuli%2C+O.%3B+Leshno%2C+M.&rft.au=Ben-Assuli%2C+O.%3B+Leshno%2C+M.&rft.date=2016&rft.volume=22&rft.issue=3&rft.pages=712%E2%80%9329&rft_id=info:doi\/10.1177%2F1460458215584203&rft_id=info:pmid\/26033468&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BoonstraBarriers10-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BoonstraBarriers10_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Boonstra, A.; Broekhuis, M. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2924334\" target=\"_blank\">\"Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions\"<\/a>. <i>BMC Health Services Research<\/i> <b>10<\/b>: 231. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6963-10-231\" target=\"_blank\">10.1186\/1472-6963-10-231<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2924334\/\" target=\"_blank\">PMC2924334<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20691097\" target=\"_blank\">20691097<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2924334\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2924334<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Barriers+to+the+acceptance+of+electronic+medical+records+by+physicians+from+systematic+review+to+taxonomy+and+interventions&rft.jtitle=BMC+Health+Services+Research&rft.aulast=Boonstra%2C+A.%3B+Broekhuis%2C+M.&rft.au=Boonstra%2C+A.%3B+Broekhuis%2C+M.&rft.date=2010&rft.volume=10&rft.pages=231&rft_id=info:doi\/10.1186%2F1472-6963-10-231&rft_id=info:pmc\/PMC2924334&rft_id=info:pmid\/20691097&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2924334&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoukemaPaper06-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RoukemaPaper06_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Roukema, J.; Los, R.K.; Bleeker, S.E. (2006). \"Paper versus computer: Feasibility of an electronic medical record in general pediatrics\". <i>Pediatrics<\/i> <b>117<\/b> (1): 15\u201321. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1542%2Fpeds.2004-2741\" target=\"_blank\">10.1542\/peds.2004-2741<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16396855\" target=\"_blank\">16396855<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Paper+versus+computer%3A+Feasibility+of+an+electronic+medical+record+in+general+pediatrics&rft.jtitle=Pediatrics&rft.aulast=Roukema%2C+J.%3B+Los%2C+R.K.%3B+Bleeker%2C+S.E.&rft.au=Roukema%2C+J.%3B+Los%2C+R.K.%3B+Bleeker%2C+S.E.&rft.date=2006&rft.volume=117&rft.issue=1&rft.pages=15%E2%80%9321&rft_id=info:doi\/10.1542%2Fpeds.2004-2741&rft_id=info:pmid\/16396855&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BowmanImpact13-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BowmanImpact13_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bowman, S. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3797550\" target=\"_blank\">\"Impact of electronic health record systems on information integrity: Quality and safety implications\"<\/a>. <i>Perspectives in Health Information Management<\/i> <b>10<\/b>: 1c. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3797550\/\" target=\"_blank\">PMC3797550<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24159271\" target=\"_blank\">24159271<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3797550\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3797550<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Impact+of+electronic+health+record+systems+on+information+integrity%3A+Quality+and+safety+implications&rft.jtitle=Perspectives+in+Health+Information+Management&rft.aulast=Bowman%2C+S.&rft.au=Bowman%2C+S.&rft.date=2013&rft.volume=10&rft.pages=1c&rft_id=info:pmc\/PMC3797550&rft_id=info:pmid\/24159271&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3797550&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NguyenElect14-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NguyenElect14_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nguyen, L.; Bellucci, E.; Nguyen, L.T. (2014). \"Electronic health records implementation: An evaluation of information system impact and contingency factors\". <i>International Journal of Medical Informatics<\/i> <b>83<\/b> (11): 779\u201396. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2014.06.011\" target=\"_blank\">10.1016\/j.ijmedinf.2014.06.011<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25085286\" target=\"_blank\">25085286<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+health+records+implementation%3A+An+evaluation+of+information+system+impact+and+contingency+factors&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Nguyen%2C+L.%3B+Bellucci%2C+E.%3B+Nguyen%2C+L.T.&rft.au=Nguyen%2C+L.%3B+Bellucci%2C+E.%3B+Nguyen%2C+L.T.&rft.date=2014&rft.volume=83&rft.issue=11&rft.pages=779%E2%80%9396&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2014.06.011&rft_id=info:pmid\/25085286&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoenEHealth13-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoenEHealth13_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moen, A.; Hackl, W.O.; Hofdijk, J. et al. (2013). \"eHealth in Europe - Status and challenges\". <i>Yearbook of Medical Informatics<\/i> <b>8<\/b>: 59\u201363. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23974549\" target=\"_blank\">23974549<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=eHealth+in+Europe+-+Status+and+challenges&rft.jtitle=Yearbook+of+Medical+Informatics&rft.aulast=Moen%2C+A.%3B+Hackl%2C+W.O.%3B+Hofdijk%2C+J.+et+al.&rft.au=Moen%2C+A.%3B+Hackl%2C+W.O.%3B+Hofdijk%2C+J.+et+al.&rft.date=2013&rft.volume=8&rft.pages=59%E2%80%9363&rft_id=info:pmid\/23974549&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OlssonEuropean04-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OlssonEuropean04_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Olsson, S.; Lymberis, A.; Whitehouse, D. (2004). \"European Commission activities in eHealth\". <i>International Journal of Circumpolar Health<\/i> <b>63<\/b> (4): 310-6. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15709306\" target=\"_blank\">15709306<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=European+Commission+activities+in+eHealth&rft.jtitle=International+Journal+of+Circumpolar+Health&rft.aulast=Olsson%2C+S.%3B+Lymberis%2C+A.%3B+Whitehouse%2C+D.&rft.au=Olsson%2C+S.%3B+Lymberis%2C+A.%3B+Whitehouse%2C+D.&rft.date=2004&rft.volume=63&rft.issue=4&rft.pages=310-6&rft_id=info:pmid\/15709306&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IakovidisEHealth08-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IakovidisEHealth08_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Iakovidis, I.; Purcarea, O. (2008). \"eHealth in Europe: From vision to reality\". <i>Studies in Health Technology and Informatics<\/i> <b>134<\/b>: 163\u20138. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18376043\" target=\"_blank\">18376043<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=eHealth+in+Europe%3A+From+vision+to+reality&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Iakovidis%2C+I.%3B+Purcarea%2C+O.&rft.au=Iakovidis%2C+I.%3B+Purcarea%2C+O.&rft.date=2008&rft.volume=134&rft.pages=163%E2%80%938&rft_id=info:pmid\/18376043&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MartinsAPop13-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MartinsAPop13_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-MartinsAPop13_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Martins, C.; Azevedo, L.F.; Ribeiro, O. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3836775\" target=\"_blank\">\"A population-based nationwide cross-sectional study on preventive health services utilization in Portugal - What services (and frequencies) are deemed necessary by patients?\"<\/a>. <i>PLOS One<\/i> <b>8<\/b> (11): e81256. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0081256\" target=\"_blank\">10.1371\/journal.pone.0081256<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3836775\/\" target=\"_blank\">PMC3836775<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24278405\" target=\"_blank\">24278405<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3836775\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3836775<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+population-based+nationwide+cross-sectional+study+on+preventive+health+services+utilization+in+Portugal+-+What+services+%28and+frequencies%29+are+deemed+necessary+by+patients%3F&rft.jtitle=PLOS+One&rft.aulast=Martins%2C+C.%3B+Azevedo%2C+L.F.%3B+Ribeiro%2C+O.&rft.au=Martins%2C+C.%3B+Azevedo%2C+L.F.%3B+Ribeiro%2C+O.&rft.date=2013&rft.volume=8&rft.issue=11&rft.pages=e81256&rft_id=info:doi\/10.1371%2Fjournal.pone.0081256&rft_id=info:pmc\/PMC3836775&rft_id=info:pmid\/24278405&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3836775&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-vanBokhovenWhy06-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-vanBokhovenWhy06_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">van Bokhoven, M.A.; Pleunis-van Empel, M.C.; Koch, H. et al. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1769380\" target=\"_blank\">\"Why do patients want to have their blood tested? A qualitative study of patient expectations in general practice\"<\/a>. <i>BMC Family Practice<\/i> <b>7<\/b>: 75. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2296-7-75\" target=\"_blank\">10.1186\/1471-2296-7-75<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1769380\/\" target=\"_blank\">PMC1769380<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17166263\" target=\"_blank\">17166263<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1769380\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1769380<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Why+do+patients+want+to+have+their+blood+tested%3F+A+qualitative+study+of+patient+expectations+in+general+practice&rft.jtitle=BMC+Family+Practice&rft.aulast=van+Bokhoven%2C+M.A.%3B+Pleunis-van+Empel%2C+M.C.%3B+Koch%2C+H.+et+al.&rft.au=van+Bokhoven%2C+M.A.%3B+Pleunis-van+Empel%2C+M.C.%3B+Koch%2C+H.+et+al.&rft.date=2006&rft.volume=7&rft.pages=75&rft_id=info:doi\/10.1186%2F1471-2296-7-75&rft_id=info:pmc\/PMC1769380&rft_id=info:pmid\/17166263&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1769380&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-vanDerWeijdenUnder02-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-vanDerWeijdenUnder02_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">van der Weijden, T.; van Bokhoven, M.A.; Dinant, G.J. et al. (2002). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1314466\" target=\"_blank\">\"Understanding laboratory testing in diagnostic uncertainty: A qualitative study in general practice\"<\/a>. <i>British Journal of General Practice<\/i> <b>52<\/b> (485): 974\u201380. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1314466\/\" target=\"_blank\">PMC1314466<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12528582\" target=\"_blank\">12528582<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1314466\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1314466<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+laboratory+testing+in+diagnostic+uncertainty%3A+A+qualitative+study+in+general+practice&rft.jtitle=British+Journal+of+General+Practice&rft.aulast=van+der+Weijden%2C+T.%3B+van+Bokhoven%2C+M.A.%3B+Dinant%2C+G.J.+et+al.&rft.au=van+der+Weijden%2C+T.%3B+van+Bokhoven%2C+M.A.%3B+Dinant%2C+G.J.+et+al.&rft.date=2002&rft.volume=52&rft.issue=485&rft.pages=974%E2%80%9380&rft_id=info:pmc\/PMC1314466&rft_id=info:pmid\/12528582&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1314466&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-vanBokhovenExploring08-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-vanBokhovenExploring08_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">van Bokhoven, M.A.; Koch, H.; Dinant, G.J. et al. (2008). \"Exploring the black box of change in improving test-ordering routines\". <i>Family Practice<\/i> <b>25<\/b> (3): 139-45. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Ffampra%2Fcmn022\" target=\"_blank\">10.1093\/fampra\/cmn022<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18535302\" target=\"_blank\">18535302<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+the+black+box+of+change+in+improving+test-ordering+routines&rft.jtitle=Family+Practice&rft.aulast=van+Bokhoven%2C+M.A.%3B+Koch%2C+H.%3B+Dinant%2C+G.J.+et+al.&rft.au=van+Bokhoven%2C+M.A.%3B+Koch%2C+H.%3B+Dinant%2C+G.J.+et+al.&rft.date=2008&rft.volume=25&rft.issue=3&rft.pages=139-45&rft_id=info:doi\/10.1093%2Ffampra%2Fcmn022&rft_id=info:pmid\/18535302&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VerstappenVariation04-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VerstappenVariation04_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Verstappen, W.H.; ter Riet, G.; Dubois, W.I. et al. (2004). \"Variation in test ordering behaviour of GPs: Professional or context-related factors?\". <i>Family Practice<\/i> <b>21<\/b> (4): 387-95. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Ffampra%2Fcmh408\" target=\"_blank\">10.1093\/fampra\/cmh408<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15249527\" target=\"_blank\">15249527<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Variation+in+test+ordering+behaviour+of+GPs%3A+Professional+or+context-related+factors%3F&rft.jtitle=Family+Practice&rft.aulast=Verstappen%2C+W.H.%3B+ter+Riet%2C+G.%3B+Dubois%2C+W.I.+et+al.&rft.au=Verstappen%2C+W.H.%3B+ter+Riet%2C+G.%3B+Dubois%2C+W.I.+et+al.&rft.date=2004&rft.volume=21&rft.issue=4&rft.pages=387-95&rft_id=info:doi\/10.1093%2Ffampra%2Fcmh408&rft_id=info:pmid\/15249527&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MainComput10-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MainComput10_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Main, C.; Moxham, T.; Wyatt, J.C. et al. (2010). \"Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: Systematic reviews of the effects and cost-effectiveness of systems\". <i>Health Technology Assessment<\/i> <b>14<\/b> (48): 1\u2013227. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3310%2Fhta14480\" target=\"_blank\">10.3310\/hta14480<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21034668\" target=\"_blank\">21034668<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Computerised+decision+support+systems+in+order+communication+for+diagnostic%2C+screening+or+monitoring+test+ordering%3A+Systematic+reviews+of+the+effects+and+cost-effectiveness+of+systems&rft.jtitle=Health+Technology+Assessment&rft.aulast=Main%2C+C.%3B+Moxham%2C+T.%3B+Wyatt%2C+J.C.+et+al.&rft.au=Main%2C+C.%3B+Moxham%2C+T.%3B+Wyatt%2C+J.C.+et+al.&rft.date=2010&rft.volume=14&rft.issue=48&rft.pages=1%E2%80%93227&rft_id=info:doi\/10.3310%2Fhta14480&rft_id=info:pmid\/21034668&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-daCostaPereiraEHealth10-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-daCostaPereiraEHealth10_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">da Costa Pereira, A.; Giest, S.; Dumortier, J.; Artmann, J. (October 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.476.4177&rep=rep1&type=pdf\" target=\"_blank\">\"eHealth Strategies - Country Brief: Portugal\"<\/a> (PDF). European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.476.4177&rep=rep1&type=pdf\" target=\"_blank\">http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.476.4177&rep=rep1&type=pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=eHealth+Strategies+-+Country+Brief%3A+Portugal&rft.atitle=&rft.aulast=da+Costa+Pereira%2C+A.%3B+Giest%2C+S.%3B+Dumortier%2C+J.%3B+Artmann%2C+J.&rft.au=da+Costa+Pereira%2C+A.%3B+Giest%2C+S.%3B+Dumortier%2C+J.%3B+Artmann%2C+J.&rft.date=October+2010&rft.pub=European+Commission&rft_id=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.476.4177%26rep%3Drep1%26type%3Dpdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-USPSTFTheGuide14-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-USPSTFTheGuide14_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">U.S. Preventive Services Task Force (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.overdrive.com\/media\/2062549\/clinical-preventive-services-2014\" target=\"_blank\">\"The Guide to Clinical Preventive Services 2014\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.overdrive.com\/media\/2062549\/clinical-preventive-services-2014\" target=\"_blank\">https:\/\/www.overdrive.com\/media\/2062549\/clinical-preventive-services-2014<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 October 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Guide+to+Clinical+Preventive+Services+2014&rft.atitle=&rft.aulast=U.S.+Preventive+Services+Task+Force&rft.au=U.S.+Preventive+Services+Task+Force&rft.date=2014&rft_id=https%3A%2F%2Fwww.overdrive.com%2Fmedia%2F2062549%2Fclinical-preventive-services-2014&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MartinsPreventive14-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MartinsPreventive14_22-0\" rel=\"external_link\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-MartinsPreventive14_22-1\" rel=\"external_link\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Martins, C.; Azevedo, L.F.; Santos, C. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4039810\" target=\"_blank\">\"Preventive health services implemented by family physicians in Portugal - A cross-sectional study based on two clinical scenarios\"<\/a>. <i>BMJ Open<\/i> <b>4<\/b> (5): e005162. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/110.1136%2Fbmjopen-2014-005162\" target=\"_blank\">110.1136\/bmjopen-2014-005162<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4039810\/\" target=\"_blank\">PMC4039810<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24861550\" target=\"_blank\">24861550<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4039810\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4039810<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Preventive+health+services+implemented+by+family+physicians+in+Portugal+-+A+cross-sectional+study+based+on+two+clinical+scenarios&rft.jtitle=BMJ+Open&rft.aulast=Martins%2C+C.%3B+Azevedo%2C+L.F.%3B+Santos%2C+C.&rft.au=Martins%2C+C.%3B+Azevedo%2C+L.F.%3B+Santos%2C+C.&rft.date=2014&rft.volume=4&rft.issue=5&rft.pages=e005162&rft_id=info:doi\/110.1136%2Fbmjopen-2014-005162&rft_id=info:pmc\/PMC4039810&rft_id=info:pmid\/24861550&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4039810&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AdminAction10-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AdminAction10_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation\" id=\"rdp-ebb-CITEREFAdministra.C3.A7.C3.A3o_Regional_de_Sa.C3.BAde_do_Norte.2C_IP2010\">Administra\u00e7\u00e3o Regional de Sa\u00fade do Norte, IP (2010), <i>Action plan 2010-2012 of the Western Oporto grouping of health centers<\/i><\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Action+plan+2010-2012+of+the+Western+Oporto+grouping+of+health+centers&rft.aulast=Administra%C3%A7%C3%A3o+Regional+de+Sa%C3%BAde+do+Norte%2C+IP&rft.au=Administra%C3%A7%C3%A3o+Regional+de+Sa%C3%BAde+do+Norte%2C+IP&rft.date=2010&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-daSa.C3.BAdeNational10-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-daSa.C3.BAdeNational10_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation\" id=\"rdp-ebb-CITEREFda_Sa.C3.BAde.2C_M.2007\">da Sa\u00fade, M. (2007), <i>National Plan for Prevention and Control of Oncological Diseases 2007\/2010<\/i><\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=National+Plan+for+Prevention+and+Control+of+Oncological+Diseases+2007%2F2010&rft.aulast=da+Sa%C3%BAde%2C+M.&rft.au=da+Sa%C3%BAde%2C+M.&rft.date=2007&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZaatLab92-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZaatLab92_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zaat, J.O.; van Eijk, J.T.; Bonte, H.A. (1992). \"Laboratory test form design influences test ordering by general practitioners in The Netherlands\". <i>Medical Care<\/i> <b>30<\/b> (3): 189\u201398. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/1538607\" target=\"_blank\">1538607<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+test+form+design+influences+test+ordering+by+general+practitioners+in+The+Netherlands&rft.jtitle=Medical+Care&rft.aulast=Zaat%2C+J.O.%3B+van+Eijk%2C+J.T.%3B+Bonte%2C+H.A.&rft.au=Zaat%2C+J.O.%3B+van+Eijk%2C+J.T.%3B+Bonte%2C+H.A.&rft.date=1992&rft.volume=30&rft.issue=3&rft.pages=189%E2%80%9398&rft_id=info:pmid\/1538607&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarthApprop01-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BarthApprop01_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barth, J.H.; Balen, A.H.; Jennings, A. (2001). \"Appropriate design of biochemistry request cards can promote the use of protocols and reduce unnecessary investigations\". <i>Annals of Clinical Biochemistry<\/i> <b>38<\/b> (Pt 6): 714\u20136. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1258%2F0004563011900957\" target=\"_blank\">10.1258\/0004563011900957<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11732657\" target=\"_blank\">11732657<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Appropriate+design+of+biochemistry+request+cards+can+promote+the+use+of+protocols+and+reduce+unnecessary+investigations&rft.jtitle=Annals+of+Clinical+Biochemistry&rft.aulast=Barth%2C+J.H.%3B+Balen%2C+A.H.%3B+Jennings%2C+A.&rft.au=Barth%2C+J.H.%3B+Balen%2C+A.H.%3B+Jennings%2C+A.&rft.date=2001&rft.volume=38&rft.issue=Pt+6&rft.pages=714%E2%80%936&rft_id=info:doi\/10.1258%2F0004563011900957&rft_id=info:pmid\/11732657&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaileyChange05-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BaileyChange05_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bailey, J.; Jennings, A.; Parapia, L. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1770878\" target=\"_blank\">\"Change of pathology request forms can reduce unwanted requests and tests\"<\/a>. <i>Journal of Clinical Pathology<\/i> <b>58<\/b> (8): 853\u20135. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fjcp.2004.023101\" target=\"_blank\">10.1136\/jcp.2004.023101<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1770878\/\" target=\"_blank\">PMC1770878<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16049288\" target=\"_blank\">16049288<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1770878\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1770878<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Change+of+pathology+request+forms+can+reduce+unwanted+requests+and+tests&rft.jtitle=Journal+of+Clinical+Pathology&rft.aulast=Bailey%2C+J.%3B+Jennings%2C+A.%3B+Parapia%2C+L.&rft.au=Bailey%2C+J.%3B+Jennings%2C+A.%3B+Parapia%2C+L.&rft.date=2005&rft.volume=58&rft.issue=8&rft.pages=853%E2%80%935&rft_id=info:doi\/10.1136%2Fjcp.2004.023101&rft_id=info:pmc\/PMC1770878&rft_id=info:pmid\/16049288&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1770878&rfr_id=info:sid\/en.wikipedia.org:Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Grammar edits were made to make the document more readable. The numerical data in the last quarter of Table 4 changed to two decimal places for no clear reason; however, it was retained as-is.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181455\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.732 seconds\nReal time usage: 0.760 seconds\nPreprocessor visited node count: 22317\/1000000\nPreprocessor generated node count: 41529\/1000000\nPost\u2010expand include size: 186199\/2097152 bytes\nTemplate argument size: 57758\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 702.358 1 - -total\n 86.05% 604.346 1 - Template:Reflist\n 74.07% 520.268 27 - Template:Citation\/core\n 66.46% 466.770 23 - Template:Cite_journal\n 9.36% 65.766 1 - Template:Infobox_journal_article\n 9.07% 63.679 49 - Template:Citation\/identifier\n 8.93% 62.687 1 - Template:Infobox\n 8.84% 62.110 2 - Template:Cite\n 5.40% 37.893 2 - Template:Cite_web\n 5.33% 37.448 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10039-0!*!0!!en!5!* and timestamp 20181214181454 and revision id 29914\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial\">https:\/\/www.limswiki.org\/index.php\/Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","45e8147ab685e78229d6e364a84469a5_images":["https:\/\/www.limswiki.org\/images\/8\/86\/Fig1_Martins_BMCMedInfoDecMak2017_17-1.gif","https:\/\/www.limswiki.org\/images\/b\/bb\/Fig2_Martins_BMCMedInfoDecMak2017_17-1.gif","https:\/\/www.limswiki.org\/images\/d\/dc\/Fig3_Martins_BMCMedInfoDecMak2017_17-1.gif"],"45e8147ab685e78229d6e364a84469a5_timestamp":1544811294,"016989f9a69b54fcd5805d9f8681d6fc_type":"article","016989f9a69b54fcd5805d9f8681d6fc_title":"Users\u2019 perspectives on a picture archiving and communication system (PACS): An in-depth study in a teaching hospital in Kuwait (Buabbas et al. 2016)","016989f9a69b54fcd5805d9f8681d6fc_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait","016989f9a69b54fcd5805d9f8681d6fc_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Users\u2019 perspectives on a picture archiving and communication system (PACS): An in-depth study in a teaching hospital in Kuwait\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nUsers\u2019 perspectives on a picture archiving and communication system (PACS): An in-depth study in a teaching hospital in KuwaitJournal\n \nJMIR Medical InformaticsAuthor(s)\n \nBuabbas, Ali Jassem; Al-Shamali, Dawood Ameer; Sharma, Prem; Haidar, Salwa; Al-Shawaf, HamzaAuthor affiliation(s)\n \nKuwait University, Mubarak AlKabeer HospitalPrimary contact\n \nEmail: ali.buabbas [at] hsc.edu.kw; Phone: 965 25319504Editors\n \nEysenbach, G.Year published\n \n2016Volume and issue\n \n4 (2)Page(s)\n \ne21DOI\n \n10.2196\/medinform.5703ISSN\n \n2291-9694Distribution license\n \nCreative Commons Attribution 2.0Website\n \nhttp:\/\/medinform.jmir.org\/2016\/2\/e21\/Download\n \nhttp:\/\/medinform.jmir.org\/2016\/2\/e21\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n\n2.1 Research questions \n\n\n3 Methods \n\n3.1 Research settings \n3.2 Study design \n3.3 Statistical analysis \n\n\n4 Results \n\n4.1 Questionnaires \n\n4.1.1 Respondent demographics \n4.1.2 Evaluation of Different Perspectives on the PACS \n4.1.3 Radiologists versus technologists \n4.1.4 Interviews \n4.1.5 Perceived system quality \n4.1.6 Perceived information quality \n4.1.7 Perceived image quality \n4.1.8 Perceived technical support services \n4.1.9 Impact of PACS on clinical practice \n4.1.10 Perceived net benefits of the PACS \n4.1.11 User satisfaction with the PACS \n4.1.12 Opinions on the PACS \n4.1.13 Expectations of the current PACS and future trends \n\n\n\n\n5 Discussion \n\n5.1 Perceptions of PACS quality, information, images, and services \n5.2 Perceptions of the PACS\u2019s impact, including net benefits and user satisfaction \n5.3 Limitations \n5.4 Conclusion \n5.5 Recommendations \n\n\n6 Acknowledgments \n7 Conflicts of interest \n8 References \n9 Abbreviations \n10 Notes \n\n\n\nAbstract \nBackground: The picture archiving and communication system (PACS) is a well-known imaging informatics application in health care organizations, specifically designed for the radiology department. Health care providers have exhibited willingness toward evaluating PACS in hospitals to ascertain the critical success and failure of the technology, considering that evaluation is a basic requirement.\nObjective: This study aimed to evaluate the success of a PACS in a regional teaching hospital of Kuwait, from users\u2019 perspectives, using information systems success criteria.\nMethods: An in-depth study was conducted by using quantitative and qualitative methods. This mixed-method study was based on: (1) questionnaires, distributed to all radiologists and technologists and (2) interviews, conducted with PACS administrators.\nResults: In all, 60 questionnaires were received from the respondents. These included 39 radiologists (75% response rate) and 21 technologists (62% response rate), with the results showing almost three-quarters (74%, 44 of 59) of the respondents rating PACS positively and as user-friendly. This study\u2019s findings revealed that the demographic data, including computer experience, was an insignificant factor, having no influence on the users\u2019 responses. The findings were further substantiated by the administrators\u2019 interview responses, which supported the benefits of PACS, indicating the need for developing a unified policy aimed at streamlining and improving the departmental workflow.\nConclusions: The PACS had a positive and productive impact on the radiologists\u2019 and technologists\u2019 work performance. They were endeavoring to resolve current problems while keeping abreast of advances in PACS technology, including teleradiology and the mobile image viewer, which is steadily increasing in usage in the Kuwaiti health system.\nKeywords: PACS evaluation, user perspective, IS success, imaging informatics, radiology \n\nIntroduction \nThe picture archiving and communication system (PACS) is a well-known imaging informatics application in health care organizations, specifically designed for the radiology department. A PACS could be defined as \u201can electronic information system (IS) used to acquire, store, transmit, and display medical images.\u201d[1] Using a PACS in a hospital has innumerable benefits at various levels.[2] At the management level, this technology has direct implications for cost reduction, rendering the film production process redundant. At the departmental level, the technology enhances productivity, as all tasks are performed digitally and swiftly. And at the clinical level, image interpretation and diagnosis become more precise and accurate.[3] For these reasons, health care organizations are increasingly adopting the PACS in their clinical radiology departments, despite the high costs, to benefit from the full advantages of using the technology. PACSs are currently being applied in many medical imaging projects around the world, such as in the United States, the United Kingdom, and Asia. However, the available literature reveals gaps with regard to the systems\u2019 effectiveness and efficiency concerning their intended use.\nThe existing literature abounds with studies evaluating the PACS.[4] However, these evaluations invariably had different focuses and objectives; for instance, there are studies on the PACS before and after the system\u2019s implementation[5], users\u2019 satisfaction[6], PACS acceptance[7], cost-effectiveness[8][9], and the system\u2019s efficiency concerning its use and in saving time.[10] The most widely used form of the PACS evaluation concerns its impact on users.[4][11][12]\nIn PACS research and practice, once the system has been adopted and implemented, it becomes imperative to evaluate the technology\u2019s effectiveness within an organization.[13] For all practical purposes, evaluation could be defined as \u201cthe process of describing the implementation of an information resource and judging its merits and worth.\u201d[14] IS deployment may invariably lead to unintended consequences, affecting the chances of the technology\u2019s success.[14] Several researchers have, therefore, recommended evaluation studies specially focused on PACSs to assess their impact in clinical practice [4,15].[4][15]\nIt is of paramount significance to investigate the success of the PACS, exploring the factors responsible for the success or failure to determine its worth clinically, based on the direct users of this system.\nThe conceptual basis of this study is focused on this: the impact of a PACS in a regional hospital in Kuwait based on specific criteria. This study is the first of its kind in Kuwait, there being a scarcity of literature in this field.\n\nResearch questions \nThe research questions were specifically as follows: (1) What impact does the PACS have on the clinical practice of radiologists and technologists in the radiology department of Mubarak Al-Kabeer Hospital? (2) Has the use of the PACS proven successful in improving the radiology department\u2019s work performance?\nThis study aimed at evaluating the success of the PACS in clinical practice, in a bid to determine the technology\u2019s merits for radiologists and technologists, including its drawbacks.\n\nMethods \nResearch settings \nThe universe of this study was Mubarak Al-Kabeer Teaching Hospital, which is one of the five regional hospitals in the State of Kuwait. Table 1 presents the site\u2019s profile. \n\n\n\n\n\n\n\nTable 1. Mubarak Al-Kabeer teaching hospital\u2019s profile\n\n\nCategories\n\nNumber\n\n\nHospital beds\n\n734\n\n\nHospitalized patients\n\n21,124\n\n\nPhysicians\n\n559\n\n\nRadiologists\n\n52\n\n\nRadiology technologists\n\n34\n\n\nPACS administrators\n\n5\n\n\nAverage number of images examined monthly\n\n32,787\n\n\n\nThis general hospital is a university-teaching hospital in Kuwait and was chosen because it is always at the forefront of development and advanced medicine. Therefore, to ensure the full advantage of the health information system (HIS), the PACS\u2019s success needed to be verified. The PACS was first introduced in the radiology department of Mubarak Al-Kabeer Hospital in 2004, marking the transition of clinical services from a film-based system to an electronic-based system. The PACS used is an off-the-shelf, Oracle-based HIS (GE Centrisity RIS i 4.2 plus, GE PACS IW 3.7.3.9 SP 3). The PACS currently has 35 workstations, with a server capacity of 64 terabytes. Radiologists use the PACS to view images through the radiology information system (RIS), which they use to report their cases. The reports generated by the RIS are then sent to the PACS, through which final reports can be sent to the HIS. The treating physician needs to submit an access request to see patients\u2019 images on the PACS. In June 2013, the PACS software was upgraded, and currently the system is fully integrated technically with the RIS and the HIS, providing the users with a secure system.\n\nStudy design \nAn in-depth study was conducted by using quantitative and qualitative methods. This mixed-method study was based on: (1) survey questionnaires, which were distributed to gather information from radiologists and technologists in the radiology department of Mubarak Al-Kabeer Hospital and (2) semi structured interviews, which were conducted to gather empirical information from the PACS administrators. Ethical approval for the study was obtained from the research department of the Ministry of Health, Kuwait.\nTo gather the responses of radiologists and technologists concerning the use of the PACS in their clinical practice, a validated questionnaire from a previous study was used.[16] The questionnaire was translated from French into English through an official translation office in Kuwait. The English version of the questionnaire was pretested with five radiologists and three technologists to ensure the suitability and usability of the questions. Accordingly, a number of amendments were made to the questionnaire. These included excluding questions that were found to be irrelevant to the technologists\u2019 use of the PACS, which comprised items that focused on retrieving, displaying, comparing, and manipulating images, including confidence level. In addition, a seven-point Likert scale was changed to five points to make it easier and more familiar for the respondents.\nIn this study, evaluating the PACS\u2019s success was based on an integrated multidimensional model, which was constructed from the model primarily developed by DeLone and McLean[17][18], later developed with two constructs added to the model, namely system continuance intention and confirmation of expectations.[16] (See Figure 1.)\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 An integrated model of picture archiving and communication system (PACS) success\n\n\n\nThe questionnaire comprised seven sections (Textbox 1) for assessing the users\u2019 perspectives on eight interrelated dimensions of the PACS success model. These included: (1) perceived system quality; (2) perceived information quality; (3) perceived service quality; (4) system usage; (5) user satisfaction; (6) perceived net benefits; (7) system continuance intention; and (8) confirmed expectations. The questionnaire was distributed to all radiologists and radiology technologists who had used the PACS in their clinical practice for the last two years.\n\n\n\n\n\n\n\nTextbox 1. Sections of the questionnaire\n\n\nSection 1: Quality of PACS\r\n\nEase of access and use\r\n \r\n* Diversity of functionalities offered by the PACS\r\n* Reliability of the hardware and software\r\n* PACS integration and compatibility with the RIS and the HIS\r\n* Security of the PACS\n\n\n\nThe data gathered through the questionnaire were complemented by conducting semi-structured interviews with PACS administrators to gain an understanding of the prevailing clinical environment, which entails them communicating with radiologists, doctors, and technologists, including providing information technology services and support.[19] Their experience further enriched the information gathered and the study\u2019s purpose.\nThe focus of the interviews was primarily similar to that of the questionnaire: to gain a deeper insight into the response patterns of the respondents. The interviews were conducted with the radiology technologists, who are responsible for administering the PACS and overseeing the RIS operations in the radiology department.\n\nStatistical analysis \nData management, analysis, and graphical presentation were carried out using the software Statistical Package for the Social Sciences (SPSS), version 22.0. The questionnaire was evaluated for internal consistency and reliability, and Cronbach alpha values were estimated for major perspectives by combining the Likert scale items for specific aspects, including quality, information, images, technical support and usage, user satisfaction, and overall opinion on the PACS. The descriptive statistics analysis generated frequencies and percentages for all the five-point Likert scale items (1 as lowest or strongly disagree and 5 as highest or strongly agree) in the questionnaire. The Likert scale data were also analyzed to find average values for overall responses and to compare the mean (\u00b1standard deviation, SD) between radiologists and technologists using t tests or nonparametric Mann-Whitney tests. The quantitative or continuous variables, age, duration of use (h), and minutes saved every day were first ascertained for normal distribution, applying the Kolmogorov-Smirnov test and were presented as mean \u00b1 SD and range for normally distributed variables and as median, range and interquartile (IQ) for skewed data. The chi-square or Fisher exact test was applied to find any association or significant difference between categorical variables. The Spearman correlation coefficient (rho) was used to find any correlations among the number of hours worked, the use of the PACS, and the minutes saved in daily practice. The two-tailed probability value P<.05 was considered statistically significant.\n\nResults \nQuestionnaires \nRespondent demographics \nThe study\u2019s overall response rate was 70%: 75% of the radiologists and 62% of the technologists of the radiology department. The study had 60 respondents: 39 radiologists (mean age = 36\u00b17.5 SD) and 21 technologists (mean age = 28\u00b110 SD). The respondents\u2019 ages varied between 20 and 60 years, with the majority (85%; 51 of 60) aged younger than 40 years. The respondents\u2019 average self-rated level of familiarity with computers was 4.8 \u00b1 1.34 (mean \u00b1 SD) on a scale of 1-7, and 41% (24 of 59) of the respondents had earlier experience with PACSs before working at this radiology department.\n\nEvaluation of Different Perspectives on the PACS \nThe overall responses on different perspectives were analyzed, and composite reliability and coefficients (Cronbach alpha) were computed and presented in Table 2, along with mean and range for each perspective. The Cronbach alpha values ranged between .73 and .96, except for one as shown in in Table 2. System quality, images produced, and services all had high (>.9) Cronbach alpha values.\nThe overall perspectives of users have been presented on the following aspects:\nSystem quality\nAlmost three-quarters (75%; 44 of 59) of the respondents rated the PACS positively and as user-friendly, with a mean of 3.28 (Table 2). Comparatively fewer (64%; 38 of 59) respondents mentioned some drawbacks of the system, such as it being temporarily out-of-service or not working, having numerous bugs, causing wait times at the workstations, and exhibiting reduced screen quality which slowed down PACS use. The majority (81%; 48 of 59) agreed that the PACS had improved the quality of services at the radiology department (mean=4.01). However, some suggestions were provided by respondents (mean=3.57) with regard to the system\u2019s improvement, including the provision of more options and investment in upgrading the visualization equipment (PC monitors).\n\n\n\n\n\n\n\nTable 2. PACS users and their responses\r\n \r\naCronbach Alpha: Measure of Internal Consistency Reliability\r\nbMean values are based on a five-point Likert scale, with 1 being the lowest and 5 being the highest.\n\n\nUser perspectives of the PACS\n\nNumber of items\n\nAlphaa\n\nMeanb\n\nRange\n\n\nQuality\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nEncouraging features\n\n15\n\n.906\n\n3.284\n\n1.567-4.033\n\n\n\n\nNon-encouraging features\n\n5\n\n.767\n\n3.000\n\n2.50-3.400\n\n\n\nInformation\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nProduce better information\n\n4\n\n.888\n\n3.754\n\n3.650-4.000\n\n\nImages\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nQuality of images produced\n\n4\n\n.910\n\n4.272\n\n4.183-4.333\n\n\n\n\nCompared to traditional films\n\n4\n\n.855\n\n4.333\n\n4.100-4.483\n\n\n\n\nConfidence in image quality\n\n2\n\n.875\n\n4.205\n\n4.154-4.256\n\n\n\n\nData adequacy\u2014access to patient data\n\n2\n\n.808\n\n3.558\n\n3.500-3.617\n\n\nTechnical support\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nReliable, prompt services\n\n7\n\n.961\n\n3.598\n\n3.483-3.683\n\n\nUse of the PACS and satisfaction\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nFrequency of PACS use\n\n5\n\n.638\n\n3.573\n\n2.583-4.000\n\n\n\n\nUser satisfaction\n\n3\n\n.887\n\n3.650\n\n3.533-3.717\n\n\nFuture use of the PACS\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nExpectations, and continuance of use\n\n3\n\n.734\n\n3.394\n\n3.233-3.483\n\n\nOverall opinion and impact of the PACS\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nImproved quality and services (benefits)\n\n21\n\n.919\n\n4.008\n\n3.169-4.390\n\n\n\nInformation quality\nIn all, 90% (53 of 59) agreed that the PACS produced better and higher-quality information (mean=3.75) that was accurate, updated, relevant, and timely. The system also provided complete patient information, including adequate access to patients\u2019 historical data (mean=3.56).\nImage quality\nThe PACS users were extremely satisfied with regard to the quality of the images produced, ease of understanding, and relevance (mean=4.27). They found that the PACS produced much better images compared to traditional film (mean=4.33).\nTechnical support and services\nThe PACS users were quite satisfied with technical support (mean=3.60) and the reliability, promptness, and dependability of services.\nUse of PACS and satisfaction\nIn all, 50% (30 of 60) of the respondents mentioned using the PACS for more than 30 hours per week (Figure 2), although a significant difference was found regarding the duration of PACS use (hours\/week) between radiologists and technologists (P<.001). A high level of user satisfaction was shown with regard to their experience in using the PACS (mean=3.65). The usage of various tools, including changing display format, retrieving, and \u201csplit screening\u201d to compare images, was found to be quite satisfactory (mean=3.57), especially among radiologists.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2 Respondents\u2019 picture archiving and communication system (PACS) use per week\n\n\n\nFuture use and expectations on PACS\nIn all, 83.9% (mean=3.39) of PACS users mentioned their expectations better than what they expected originally and showed intention to continue using PACS.\nOverall opinions and impact of PACS\nBased on 21 different statements, 93% (56 of 60) of the PACS users showed consensus on various aspects of the system\u2019s benefits and effectiveness (mean=4.01), and the mean was significantly higher for technologists as compared with radiologists (4.22 vs 3.89). Furthermore, the results showed that 80% (48 of 60) of the PACS users reported saving more than 30 minutes of their practice time each day, whereas 38% (23 of 60) mentioned saving more than an hour each day.\n\nRadiologists versus technologists \nTable 3 summarizes the comparison between radiologists\u2019 and technologists\u2019 responses with regard to their perspectives concerning the PACS. The mean values were significantly higher for the technologists as compared with the radiologists, especially concerning quality, information, patient data, technical support, and overall opinion on impact of the PACS (P<.05). Both professionals showed the highest level of satisfaction (mean >4) with regard to image produced, and their overall opinions on PACS demonstrated improved quality and services (radiologist 3.9 and technologists 4.2).\n\n\n\n\n\n\n\nTable 3. Radiologists\u2019 and technologists\u2019 responses\r\n \r\naMean values are based on a five-point Likert scale, with 1 being the lowest and 5 being the highest.\r\nbTechnologists were not asked this question, as the decision on image quality lies on radiologists.\n\n\nUser perspectives of the PACS\n\nRadiologists (n=39)\n\nTechnologists (n=21)\n\nP value\n\n\nMeana\n\nSD\n\nMeana\n\nSD\n\n\nQuality\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nEncouraging features\n\n3.109\n\n0.559\n\n3.733\n\n0.528\n\n.006\n\n\n\n\nNon-encouraging features\n\n3.070\n\n0.693\n\n3.333\n\n0.563\n\n.244\n\n\nInformation\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nProduce better information\n\n3.539\n\n0.830\n\n4.155\n\n0.886\n\n.007\n\n\nImages\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nQuality of images produced\n\n4.188\n\n0.692\n\n4.429\n\n0.598\n\n.186\n\n\n\n\nCompared to traditional films\n\n4.436\n\n0.622\n\n4.143\n\n0.705\n\n.083\n\n\n\n\nConfidence in image qualityb\n\n4.205\n\n0.704\n\n---\n\n---\n\n---\n\n\n\n\nData adequacy\u2014access to patient data\n\n3.295\n\n1.074\n\n4.048\n\n0.879\n\n.005\n\n\nTechnical support\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nReliable, prompt services\n\n3.396\n\n1.080\n\n3.973\n\n0.600\n\n.029\n\n\nUse of the PACS and satisfaction\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nFrequency of PACS use\n\n3.585\n\n0.760\n\n3.552\n\n0.819\n\n.963\n\n\n\n\nUser satisfaction\n\n3.556\n\n0.863\n\n3.825\n\n0.611\n\n.144\n\n\nFuture use of the PACS\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nExpectations, and continuance of use\n\n3.282\n\n0.867\n\n3.603\n\n0.629\n\n.140\n\n\nOverall opinion and impact of the PACS\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nImproved quality and services (benefits)\n\n3.892\n\n0.623\n\n4.218\n\n0.427\n\n.050\n\n\n\nIn total, 49% (19 of 39) of the radiologists mentioned saving more than 60 minutes every day, as compared to 19% (4 of 21) of the technologists (P=.048) (Figure 3).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 3 Respondents\u2019 minutes saved per day\n\n\n\nDuring use of the PACS, both professional groups reported significant savings in the working time for different modalities, though with much variation (the Kolmogorov\u2013Smirnov tests showed a skewed distribution); the median and interquartile have been presented in Figure 4 as a box-plot. The maximum number of minutes saved was 52 minutes (median time) by radiologists in magnetic resonance imaging and 50 minutes by technologists in radiography.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 4 Average minutes (median with interquartile range) saved per day by picture archiving and communication system (PACS) users in different modalities\n\n\n\nA significant positive correlation was observed between the number of hours using the PACS and the minutes saved in daily practice since the introduction of the PACS (r=0.27, P=.037).\nThe level of prior familiarity with computers was found to be similar between the radiologists (4.84\u00b11.34 SD) and the technologists (4.71\u00b11.35 SD) and did not make any significant difference either in the average duration (hours\/week) of working with the PACS or the time saved (minutes\/day) during practice.\nThe results of the open-ended questions showed that 24% (9 of 38) of the radiologists and 33% (7 of 21) of the technologists stated that storing, retrieving, and comparing images were the most positive elements associated with the use of the PACS. By contrast, 33% (13 of 39) of the radiologists and 43% (9 of 21) of the technologists stated that frequent glitches were the most negative element associated with the PACS.\nOverall, the study\u2019s findings revealed that both the radiologists and the technologists perceived the adoption of the PACS positively. The mean scores were mostly above 3 or 4 on a scale of 1-5. The mean scores for image quality and information produced were 4.3 and 3.8, respectively. The users seemed quite satisfied with the services and technical support, with a mean score of 3.6 and showed satisfaction in working with the PACS (mean=3.65). The PACS users clearly mentioned improved services and quality since the system came into practice, with a mean score of 4.\n\nInterviews \nThe opinions of the PACS administrators were obtained by using the interview method, for which a series of semi-structured questions on specific themes (Textbox 1) provided the basis for soliciting information.\nAt the time of the interviews, Mubarak Al-Kabeer Hospital had five PACS administrators. For scheduling the interviews, requisite permission was taken from the head of the radiology department, and interview sessions were arranged with the staff during their respective work breaks, over a five-day period. Each interview session lasted approximately 50 minutes. The interviews were transcribed, and the responses were coded and analyzed using thematic analysis.\nThe interview results showed that all the interviewees had a BSc degree in radiological sciences, with their ages ranging between 25 and 35 years, and each having work experience of 2-5 years in PACS administration. Of the interviewees, only three had undertaken an introductory training program abroad on PACS use and management.\n\nPerceived system quality \nThe interview responses confirmed that the PACS provided easy access to authorized users, each with a user identification (ID) number and password, thereby providing a secured workspace depending on the user\u2019s position. For instance, a radiology technologist\u2019s access is limited to only viewing the reporting screen, with no authorization to change or manipulate it, thus preserving the data, with no hacking or security problems ever encountered or reported.\nThe interviewees unanimously agreed that the PACS was user friendly and hassle free in its functionality. In one of the interviewee\u2019s words, \u201cWe haven\u2019t experienced any complaints from radiologists regarding the clarity of the PACS\u2019s features, or any difficulties in moving between its functions,\u201d further adding that training in the PACS should be a prerequisite before its use.\nThe participants also endorsed the reliability and consistency of the existing hardware, including computer systems, networks, and printers, with the software used. The interviews further revealed that the PACS was fully integrated and compatible with the RIS and the HIS, although the workflow did not follow the planned process, as Figure 5 demonstrates. According to one interviewee, \u201cThe real mistakes are not coming from the PACS but from humans, so they\u2019re human errors.\u201d The interview responses also highlighted that the problems associated with PACS integration and compatibility with the RIS and the HIS were the result of disorganized workflow, as shown in Figure 5.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 5 Workstations where electronic registrations of patients failed through the health information system (HIS) and radiology information system (RIS)\n\n\n\nFigure 5 (above) illustrates the workstations where electronic registration of patients through the HIS and the RIS failed due to receptionist errors such as: (1) no data entry into HIS, manual registration in the RIS; (2) failure of communication between the HIS and the RIS, manual registration in the RIS; (3) and incorrect registration at reception, manual registration in the RIS (Figure 6).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 6 Manual registration of patients through the health information system (HIS) and radiology information system (RIS)\n\n\n\nThe manual registration at these three workstations resulted in: (1) a lack of direct access to patients\u2019 imaging results through the HIS; (2) the creation of multiple PACS numbers for the same patient, making it difficult to retrieve previous reports for comparison, as well as the loss of patient data; and (3) delayed patient case management due to a failure in the rapid delivery of results.\n\nPerceived information quality \nThe interviewees agreed that the PACS provided a standard format for the acquisition of accurate and complete information, together with images, concerning the patients\u2019 medical cases, including their name, age, gender, national identification number, medical record number, and medical history. The lapses that occurred in the recorded information were attributed to the registration staff of the diagnostic radiology department because of their noncompliance to instructions, which resulted in incomplete data records of patients at the time of registration.\n\nPerceived image quality \nThe interview responses indicated that one of the main roles of PACS administrators was to ensure that the images were transferred and displayed with clarity to facilitate studying and reporting. The participants further confirmed that \u201cWe experience hangs in the images in PACS, but at an acceptable rate\u201d and no complaints were mentioned concerning image manipulation and management.\n\nPerceived technical support services \nAs the interviewees mentioned, the main IT support is delivered through the company that sold the PACS. This usually happens when the PACS administrators face a technical problem that can only be solved through the main IT support at the company. Thus, the PACS administrators asked to have some power to authorize them to solve the technical issues within the radiology department. One of the interviewees stated: \u201c\u2026even when we want to connect a new printer to the PACS, we have to call the main IT support to perform this function for us.\u201d However, all of the interviewees complimented the IT support services at the company for their prompt responses to any technical issues.\n\nImpact of PACS on clinical practice \nPACS has an impact on the clinical practice of radiologists and technologists, as shown in the interviews' results.\n\nPerceived net benefits of the PACS \nFrom the interviews, it was easy to see that the PACS has increased users\u2019 productivity in comparison to the traditional filming system by minimizing their effort and time. In addition, the retaking of images is not required, as the PACS facilitates image storage and retrieval faster and over a longer period. \u201cWe are happy with the PACS\u2019s benefits,\u201d reported one interviewee, although the system has slowed in speed due to the huge number of cases, with the intervening procedures passing through several modalities, such as computed tomography and magnetic resonance imaging. There is also the possibility of missed images, especially concerning unknown IDs, although these could be traced using the patient\u2019s civil ID, the patient\u2019s PACS ID, or the excision ID of images.\n\nUser satisfaction with the PACS \nAll the interviewees were apparently satisfied with the PACS; however, the technology-associated problems need to be addressed to optimize the system\u2019s versatility and performance.\n\nOpinions on the PACS \nOverall, from the interviews, the responses revealed that as long as the image is electronically collected, stored, and communicated to another system successfully, the productively of work will be increased, diagnosis will be precisely performed, the patient will be treated accurately and quickly, and health services will be improved.\n\nExpectations of the current PACS and future trends \nThe interviewees expressed satisfaction in using the PACS system but also highlighted the need for resolving the current problems, as well as to keep abreast of the latest advances in PACS operations, to meet the growing demands of the Kuwaiti health system. The emerging requirements for potential trends in the future concern the areas of: (1) teleradiology services (for radiologists to use the PACS anywhere and anytime); (2) a mobile image viewer for faster accessibility to images; (3) speech recognition functions; (4) computer-assisted diagnosis (CAD); (5) advanced training; and (6) recruiting health informatics graduates to support the PACS administrators.\n\nDiscussion \nIn general, the study\u2019s findings revealed that the PACS has had a productive impact on the staff\u2019s clinical practice. Despite some of the technical limitations of the infrastructure, most of the respondents rated the system positively and as user-friendly. The findings showed that the technologists were more satisfied than the radiologists were with using the PACS. Interestingly, there was a significant relationship between the perceived benefits of the PACS and the willingness of users to continue using it. It was also noteworthy that the problems associated with the PACS\u2019s integration with the RIS were the result of disorganized workflow.\nThe results of the study revealed that the users\u2019 demographic data, including computer experience, had no influence on their response patterns, being insignificant determinants of their predilection or preference for the PACS in enhancing their work efficiency. These findings were consistent with the study\u2019s results on PACS acceptance[7], but contradicted with the results of earlier studies that reported the significant influence of age and gender on users\u2019 choices concerning information technology, such as computer use patterns[20][21], particularly to adopting PACS.[3][8]\n\nPerceptions of PACS quality, information, images, and services \nThe study further revealed that both the radiologists and the technologists were satisfied with the quality of information and images produced and had positive views regarding the use of this technology. The PACS offered the users with the requisite information on a medical case and facilitated the accomplishment of several functions with efficiency and ease in producing high-quality images with precision and clarity. This positive relationship found between users' satisfaction and quality of information and images produced by PACS was consistent with the findings of previous studies.[1][22] The results of the interviews further complemented these findings, with no mention of lost images posing a major problem, due to successful image retrieval by PACS administrators.\nThe study found that the technologists were more satisfied than the radiologists concerning their current PACS use, attributing their satisfaction to two reasons, which had been confirmed in previous studies[2][6][23]: the technologists achieved their core objectives of using the PACS, including image access, storage, and retrieval; and (2) the radiologists looked beyond these features for additional facilities and functions, such as the PACS being packaged with CAD, teleradiology, or speech recognition functions. As the radiologists had been using the PACS far longer than the technologists had, their understanding and familiarity with the PACS appeared to be relatively higher.\nConcerning the quality of the services offered to support PACS technically, the findings showed that both users were satisfied with the technical support provided with regard to the promptness, reliability, and dependability of the services. However, the results of the interviews revealed that the radiologists and the technologists encountered organizational and infrastructure deficiencies. On the technical level, there was frequent breakdown of the system during rush hours; and on organizational level, there was negligence of some receptionists in recording patients\u2019 information from the RIS to the HIS. Interestingly, the respondents still showed satisfaction in confirming the benefits of the PACS over conventional radiology despite some deficiencies, as reportedly addressed in previous studies.[3][22]\n\nPerceptions of the PACS\u2019s impact, including net benefits and user satisfaction \nRegarding the PACS\u2019s net benefits, the findings demonstrated that both the radiologists and the technologists had used the PACS to enhance their work productivity with ease due to the swift storage, retrieval, and transfer of images along with reports. These findings were consistent with those reported in previous evaluative studies on the impact of PACS[6][24], confirming that work productivity in regard to the given effort, time, and accuracy of reporting has obviously been improved. Furthermore, the PACS\u2019s benefits were found to have direct implications for user satisfaction, affecting their continued use of the PACS in the future.[16] These previous studies concluded that the more the users agreed with a PACS\u2019s effectiveness in their work, the more they were satisfied and willing to continue using it. The findings of the interviews further confirmed that both types of users benefited from the PACS\u2019s advantages, expressing their readiness toward the technology\u2019s continuous use while looking ahead for additional functions, without deficiencies, which coincided with other studies.[25][26][27]\n\nLimitations \n(1) This study was limited to radiologists and radiology technologists and did not involve other health care providers who are responsible for receiving patients\u2019 reports and images. Hence, there is a need for further research that would substantiate the study\u2019s findings by involving other stakeholders using the PACS facility, for the purpose of comparing research outcomes and enhancing the study\u2019s value. (2) The study also did not include socioeconomic and cultural factors, which are significant predictors of IT adoption in the Arab world[7][28][29] in comparison to Western countries. However, the respondents\u2019 willingness to use the PACS was a positive indicator of the technology\u2019s versatility, efficiency, and continuous use. (3) As the study was confined to one general hospital in Kuwait, there is a definitive need for future studies to enhance the study\u2019s scope by including other hospitals where PACSs are being used, for comparative purposes. (4) The study used specific criteria in evaluating IS success; hence, there is a need for using different models and tools for exploring and assessing PACSs and RISs from different dimensions.\n\nConclusion \nEvaluating the applications of imaging informatics, such as PACSs, in hospitals is crucial to ensure the successful implementation of the applications, to identify the systems\u2019 strengths and weaknesses during operation, and to provide the opportunity for further improvements, strengthening the positive elements and minimizing drawbacks.\nThe evaluation of the existing PACS at Mubarak Al-Kabeer teaching hospital led to the successful assessment of the technology\u2019s implications, based on which the study\u2019s conclusions are summarized: (1) the PACS exhibited a positive impact on the radiologists and the technologists in the diagnostic radiology department, significantly enhancing their work efficiency and productivity. Therefore, the impact of the technology was particularly visible in the context of its ability to store and retrieve images quickly, enabling the users to accomplish their tasks swiftly. In addition, the system facilitated the addition of an image to a report, expediting communication with another location with a keystroke; (2) the main concern reported by all the users was the frequent breakdown during rush hours at busy workstations, due to infrastructure deficiency; (3) both the technologists and the radiologists indicated the need for a more-advanced PACS in response to the growing demand of teleradiology, mobile image viewing, and voice recognition features; and (4) evaluating PACS\u2019s success is not confined to the technology itself but also concerns organizational and human factors that could limit the full integration with HIS.\n\nRecommendations \nTo improve the work on the current PACS and overcome the deficiencies, the following recommendations could be considered at Mubarak Al-Kabeer general hospital: (1) Enhance the capacity of existing servers to accommodate the huge amount of data generated from the massive inflow of patients. (2) Develop an internal policy to facilitate the coordination with the hospital management for organizing hospital workflow with efficiency. This policy should be followed carefully by the department staff for achieving the full benefits of the PACS\u2019s integration with the HIS and the RIS. (3) Offer advanced training courses for fully using the PACS\u2019s functions. (4) Keep a forward-looking eye on future trends of PACS, including teleradiology services, mobile images viewing, speech recognition, and CAD. (5) Hire health informatics specialists for providing the requisite administrative support on account of their knowledge in the field.\n\nAcknowledgments \nSpecial thanks go to all the staff at the radiology department of Mubarak Al-Kabeer Teaching Hospital in Kuwait for their efforts and contributions in making this study successful.\n\nConflicts of interest \nNone declared.\n\nReferences \n\n\n\u2191 1.0 1.1 Tzeng, W.S.; Kuo, K.M.; Lin, H.W.; Chen, T.Y. (2013). \"A socio-technical assessment of the success of Picture Archiving and Communication Systems: The radiology technologist's perspective\". BMC Medical Informatics and Decision Making 13: 109. doi:10.1186\/1472-6947-13-109. PMC PMC3849362. PMID 24053458. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849362 .   \n\n\u2191 2.0 2.1 Vest, J.R.; Jung, H.Y.; Ostrovsky, A. et al. (2015). \"Image Sharing Technologies and Reduction of Imaging Utilization: A Systematic Review and Meta-analysis\". Journal of the American College of Radiology 12 (12 Pt B): 1371-1379.e3. doi:10.1016\/j.jacr.2015.09.014. PMC PMC4730956. PMID 26614882. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4730956 .   \n\n\u2191 3.0 3.1 3.2 Duyck, P.; Pynoo, B.; Devolder, P. et al. (2010). \"Monitoring the PACS implementation process in a large university hospital: Discrepancies between radiologists and physicians\". Journal of Digital Imaging 23 (1): 73\u201380. doi:10.1007\/s10278-008-9163-7. PMC PMC3043749. PMID 18956231. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3043749 .   \n\n\u2191 4.0 4.1 4.2 Hains, I.M.; Georgiou, A.; Westbrook, J.I. (2012). \"The impact of PACS on clinician work practices in the intensive care unit: a systematic review of the literature\". JAMIA 19 (4): 506\u201313. doi:10.1136\/amiajnl-2011-000422. PMC PMC3384105. PMID 22323392. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3384105 .   \n\n\u2191 Bryan, S.; Weatherburn, G.C.; Watkins, J.R.; Buxton, M.J. (1999). \"The benefits of hospital-wide picture archiving and communication systems: A survey of clinical users of radiology services\". The British Journal of Radiology 72 (857): 469-78. doi:10.1259\/bjr.72.857.10505012. PMID 10505012.   \n\n\u2191 6.0 6.1 6.2 Tan, S.L.; Lewis, R.A. (2010). \"Picture archiving and communication systems: A multicentre survey of users experience and satisfaction\". European Journal of Radiology 75 (3): 406\u201310. doi:10.1016\/j.ejrad.2009.05.008. PMID 19523778.   \n\n\u2191 7.0 7.1 7.2 Aldosari, B. (2012). \"User acceptance of a picture archiving and communication system (PACS) in a Saudi Arabian hospital radiology department\". BMC Medical Informatics and Decision Making 12: 44. doi:10.1186\/1472-6947-12-44. PMC PMC3423046. PMID 22640490. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3423046 .   \n\n\u2191 8.0 8.1 Bryan, S.; Weatherburn, G.; Buxton, M. et al. (1999). \"Evaluation of a hospital picture archiving and communication system\". Journal of Health Services Research and Policy 4 (4): 204-9. doi:10.1177\/135581969900400405. PMID 10623035.   \n\n\u2191 Moodley, I.; Moodley, S. (2015). \"A comparative cost analysis of picture archiving and communications systems (PACS) versus conventional radiology in the private sector\". South African Journal of Radiology 19 (1): 634-641. doi:10.4102\/sajr.v19i1.634.   \n\n\u2191 Mackinnon, A.D.; Billington, R.A.; Adam, E.J. et al. (2008). \"Picture archiving and communication systems lead to sustained improvements in reporting times and productivity: Results of a 5-year audit\". Clinical Radiology 63 (7): 796-804. doi:10.1016\/j.crad.2007.12.015. PMID 18555038.   \n\n\u2191 van de Wetering, R.; Batenburg, R.; Versendaal, J. et al. (2006). \"A balanced evaluation perspective: Picture archiving and communication system impacts on hospital workflow\". Journal of Digital Imaging 19 (Suppl 1): 10. doi:10.1007\/s10278-006-0628-2. PMC PMC3045172. PMID 16763932. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3045172 .   \n\n\u2191 Creswick, N.; Hains, I.M.; Westbrook, J.I. (2011). \"Innovation in intensive care nursing work practices with PACS\". Studies in Health Technology and Informatics 169: 402\u20136. doi:10.3233\/978-1-60750-806-9-402. PMID 21893781.   \n\n\u2191 van de Wetering, R.; Batenburg, R. (2014). \"Towards a theory of PACS deployment: An integrative PACS maturity framework\". Journal of Digital Imaging 27 (3): 337\u201350. doi:10.1007\/s10278-013-9671-y. PMC PMC4026462. PMID 24448919. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4026462 .   \n\n\u2191 14.0 14.1 Friedman, C.P.; Wyatt, J. (2006). Evaluation Methods in Biomedical Informatics. Springer-Verlag New York. pp. 386. doi:10.1007\/0-387-30677-3. ISBN 9780387258898.   \n\n\u2191 Mansoori, B.; Erhard, K.K.; Sunshine, J.L. (2012). \"Picture Archiving and Communication System (PACS) implementation, integration & benefits in an integrated health system\". Academic Radiology 19 (2): 229\u201335. doi:10.1016\/j.acra.2011.11.009. PMID 22212425.   \n\n\u2191 16.0 16.1 16.2 Par\u00e9, G.; Lepanto, L.; Aubry, D.; Sicotte, C. (2005). \"Toward a multidimensional assessment of picture archiving and communication system success\". International Journal of Technology Assessment in Health Care 21 (4): 471\u20139. doi:10.1017\/S0266462305050658. PMID 16262970.   \n\n\u2191 DeLone, W.H.; McLean, E.R. (1992). \"Information Systems Success: The Quest for the Dependent Variable\". Information Systems Research 3 (1): 60\u201395. doi:10.1287\/isre.3.1.60.   \n\n\u2191 DeLone, W.H.; McLean, E.R. (2003). \"The DeLone and McLean Model of Information Systems Success: A Ten-Year Update\". Journal of Management Information Systems 19 (4): 9\u201330. doi:10.1080\/07421222.2003.11045748.   \n\n\u2191 Joshi, V.; Narra, V.R.; Joshi, K. (2014). \"PACS administrators' and radiologists' perspective on the importance of features for PACS selection\". Journal of Digital Imaging 27 (4): 486-95. doi:10.1007\/s10278-014-9682-3. PMC PMC4090402. PMID 24744278. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4090402 .   \n\n\u2191 Ward, R.; Stevens, C.; Brentnall, P.; Briddon, J. (2008). \"The attitudes of health care staff to information technology: A comprehensive review of the research literature\". Health Information and Libraries Journal 25 (2): 81-97. doi:10.1111\/j.1471-1842.2008.00777.x. PMID 18494643.   \n\n\u2191 Adeleke, I.; Adejoke, A.; Achinbee, M. et al. (2015). \"ICT knowledge, Utilization and Perception among Healthcare Providers at National Hospital Abuja, Nigeria\". American Journal of Health Research 3 (1-1): 47-53. doi:10.11648\/j.ajhr.s.2015030101.173.   \n\n\u2191 22.0 22.1 Sicotte, C.; Par\u00e9, G.; Bini, K.K. et al. (2010). \"Virtual organization of hospital medical imaging: A user satisfaction survey\". Journal of Digital Imaging 23 (6): 689-700. doi:10.1007\/s10278-009-9220-x. PMC PMC3046687. PMID 19588196. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3046687 .   \n\n\u2191 Folio, L.R.; Nelson, C.J.; Benjamin, M. et al. (2015). \"Quantitative Radiology Reporting in Oncology: Survey of Oncologists and Radiologists\". American Journal of Roentgenology 205 (3): W233-43. doi:10.2214\/AJR.14.14054. PMID 26295661.   \n\n\u2191 Sevenster, M.; Travis, A.R.; Ganesh, R.K. et al. (2015). \"Improved efficiency in clinical workflow of reporting measured oncology lesions via PACS-integrated lesion tracking tool\". American Journal of Roentgenology 204 (3): 576-83. doi:10.2214\/AJR.14.12915. PMID 25714288.   \n\n\u2191 Hsiao, J.L.; Chen, R.F. (2016). \"Critical factors influencing physicians' intention to use computerized clinical practice guidelines: An integrative model of activity theory and the technology acceptance model\". BMC Medical Informatics and Decision Making 16: 3. doi:10.1186\/s12911-016-0241-3. PMC PMC4715302. PMID 26772169. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4715302 .   \n\n\u2191 Liu, C.F.; Cheng, T.J. (2015). \"Exploring critical factors influencing physicians' acceptance of mobile electronic medical records based on the dual-factor model: A validation in Taiwan\". BMC Medical Informatics and Decision Making 15: 4. doi:10.1186\/s12911-014-0125-3. PMC PMC4333263. PMID 25889506. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4333263 .   \n\n\u2191 Floyd, D.M.; Trepp, E.R.; Ipaki, M.; Ng, C.K. (2015). \"Study of Radiologic Technologists' Perceptions of Picture Archiving and Communication System (PACS) Competence and Educational Issues in Western Australia\". Journal of Digital Imaging 28 (3): 315\u201322. doi:10.1007\/s10278-014-9765-1. PMC PMC4441692. PMID 25561072. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4441692 .   \n\n\u2191 Ray, K.N.; Felmet, K.A.; Hamilton, M.F. et al. (2017). \"Clinician Attitudes Toward Adoption of Pediatric Emergency Telemedicine in Rural Hospitals\". Pediatric Emergency Care 33 (4): 250-257. doi:10.1097\/PEC.0000000000000583. PMID 26785087.   \n\n\u2191 Straub, D.; Loch, K.; Hill, C. (2001). \"Transfer of Information Technology to the Arab World: A Test of Cultural Influence Modeling\". Journal of Global Information Management 9 (4): 6\u201328. doi:10.4018\/jgim.2001100101.   \n\n\nAbbreviations \nCAD: computer-assisted diagnosis\nHIS: health information system\nID: identification\nIS: information system\nPACS: picture archiving and communication System\nRIS: radiology information system\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID was missing and was added to make the reference more useful. \nPer the distribution agreement, the following copyright information is also being added: \n\u00a9Ali Jassem Buabbas, Dawood Ameer Al-Shamali, Prem Sharma, Salwa Haidar, Hamza Al-Shawaf. Originally published in JMIR Medical Informatics (http:\/\/medinform.jmir.org), 15.06.2016.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\">https:\/\/www.limswiki.org\/index.php\/Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on imaging informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 31 May 2017, at 20:35.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,817 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","016989f9a69b54fcd5805d9f8681d6fc_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Users\u2019_perspectives_on_a_picture_archiving_and_communication_system_PACS_An_in-depth_study_in_a_teaching_hospital_in_Kuwait skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Users\u2019 perspectives on a picture archiving and communication system (PACS): An in-depth study in a teaching hospital in Kuwait<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: The <a href=\"https:\/\/www.limswiki.org\/index.php\/Picture_archiving_and_communication_system\" title=\"Picture archiving and communication system\" target=\"_blank\" class=\"wiki-link\" data-key=\"523b73ff51fa83663dc0b1d59e6d0f05\">picture archiving and communication system<\/a> (PACS) is a well-known <a href=\"https:\/\/www.limswiki.org\/index.php\/Imaging_informatics\" title=\"Imaging informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"fc0ae6a154d8896767defefdb6d14d0e\">imaging informatics<\/a> application in health care organizations, specifically designed for the radiology department. Health care providers have exhibited willingness toward evaluating PACS in hospitals to ascertain the critical success and failure of the technology, considering that evaluation is a basic requirement.\n<\/p><p><b>Objective<\/b>: This study aimed to evaluate the success of a PACS in a regional teaching hospital of Kuwait, from users\u2019 perspectives, using information systems success criteria.\n<\/p><p><b>Methods<\/b>: An in-depth study was conducted by using quantitative and qualitative methods. This mixed-method study was based on: (1) questionnaires, distributed to all radiologists and technologists and (2) interviews, conducted with PACS administrators.\n<\/p><p><b>Results<\/b>: In all, 60 questionnaires were received from the respondents. These included 39 radiologists (75% response rate) and 21 technologists (62% response rate), with the results showing almost three-quarters (74%, 44 of 59) of the respondents rating PACS positively and as user-friendly. This study\u2019s findings revealed that the demographic data, including computer experience, was an insignificant factor, having no influence on the users\u2019 responses. The findings were further substantiated by the administrators\u2019 interview responses, which supported the benefits of PACS, indicating the need for developing a unified policy aimed at streamlining and improving the departmental workflow.\n<\/p><p><b>Conclusions<\/b>: The PACS had a positive and productive impact on the radiologists\u2019 and technologists\u2019 work performance. They were endeavoring to resolve current problems while keeping abreast of advances in PACS technology, including teleradiology and the mobile image viewer, which is steadily increasing in usage in the Kuwaiti health system.\n<\/p><p><b>Keywords<\/b>: PACS evaluation, user perspective, IS success, imaging informatics, radiology \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>The picture archiving and communication system (PACS) is a well-known imaging informatics application in health care organizations, specifically designed for the radiology department. A PACS could be defined as \u201can electronic information system (IS) used to acquire, store, transmit, and display medical images.\u201d<sup id=\"rdp-ebb-cite_ref-TzengASocio13_1-0\" class=\"reference\"><a href=\"#cite_note-TzengASocio13-1\" rel=\"external_link\">[1]<\/a><\/sup> Using a PACS in a hospital has innumerable benefits at various levels.<sup id=\"rdp-ebb-cite_ref-VestImage15_2-0\" class=\"reference\"><a href=\"#cite_note-VestImage15-2\" rel=\"external_link\">[2]<\/a><\/sup> At the management level, this technology has direct implications for cost reduction, rendering the film production process redundant. At the departmental level, the technology enhances productivity, as all tasks are performed digitally and swiftly. And at the clinical level, image interpretation and diagnosis become more precise and accurate.<sup id=\"rdp-ebb-cite_ref-DuyckMonitoring10_3-0\" class=\"reference\"><a href=\"#cite_note-DuyckMonitoring10-3\" rel=\"external_link\">[3]<\/a><\/sup> For these reasons, health care organizations are increasingly adopting the PACS in their clinical radiology departments, despite the high costs, to benefit from the full advantages of using the technology. PACSs are currently being applied in many medical imaging projects around the world, such as in the United States, the United Kingdom, and Asia. However, the available literature reveals gaps with regard to the systems\u2019 effectiveness and efficiency concerning their intended use.\n<\/p><p>The existing literature abounds with studies evaluating the PACS.<sup id=\"rdp-ebb-cite_ref-HainsTheImpact12_4-0\" class=\"reference\"><a href=\"#cite_note-HainsTheImpact12-4\" rel=\"external_link\">[4]<\/a><\/sup> However, these evaluations invariably had different focuses and objectives; for instance, there are studies on the PACS before and after the system\u2019s implementation<sup id=\"rdp-ebb-cite_ref-BryanTheBenefits99_5-0\" class=\"reference\"><a href=\"#cite_note-BryanTheBenefits99-5\" rel=\"external_link\">[5]<\/a><\/sup>, users\u2019 satisfaction<sup id=\"rdp-ebb-cite_ref-TanPicture10_6-0\" class=\"reference\"><a href=\"#cite_note-TanPicture10-6\" rel=\"external_link\">[6]<\/a><\/sup>, PACS acceptance<sup id=\"rdp-ebb-cite_ref-AldosariUser12_7-0\" class=\"reference\"><a href=\"#cite_note-AldosariUser12-7\" rel=\"external_link\">[7]<\/a><\/sup>, cost-effectiveness<sup id=\"rdp-ebb-cite_ref-BryanEval99_8-0\" class=\"reference\"><a href=\"#cite_note-BryanEval99-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MoodleyAComp15_9-0\" class=\"reference\"><a href=\"#cite_note-MoodleyAComp15-9\" rel=\"external_link\">[9]<\/a><\/sup>, and the system\u2019s efficiency concerning its use and in saving time.<sup id=\"rdp-ebb-cite_ref-MackinnonPicture08_10-0\" class=\"reference\"><a href=\"#cite_note-MackinnonPicture08-10\" rel=\"external_link\">[10]<\/a><\/sup> The most widely used form of the PACS evaluation concerns its impact on users.<sup id=\"rdp-ebb-cite_ref-HainsTheImpact12_4-1\" class=\"reference\"><a href=\"#cite_note-HainsTheImpact12-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VanDeWeteringABalanced06_11-0\" class=\"reference\"><a href=\"#cite_note-VanDeWeteringABalanced06-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CreswickInnovation11_12-0\" class=\"reference\"><a href=\"#cite_note-CreswickInnovation11-12\" rel=\"external_link\">[12]<\/a><\/sup>\n<\/p><p>In PACS research and practice, once the system has been adopted and implemented, it becomes imperative to evaluate the technology\u2019s effectiveness within an organization.<sup id=\"rdp-ebb-cite_ref-VanDeWeteringTowards14_13-0\" class=\"reference\"><a href=\"#cite_note-VanDeWeteringTowards14-13\" rel=\"external_link\">[13]<\/a><\/sup> For all practical purposes, evaluation could be defined as \u201cthe process of describing the implementation of an information resource and judging its merits and worth.\u201d<sup id=\"rdp-ebb-cite_ref-FriedmanEvaluation06_14-0\" class=\"reference\"><a href=\"#cite_note-FriedmanEvaluation06-14\" rel=\"external_link\">[14]<\/a><\/sup> IS deployment may invariably lead to unintended consequences, affecting the chances of the technology\u2019s success.<sup id=\"rdp-ebb-cite_ref-FriedmanEvaluation06_14-1\" class=\"reference\"><a href=\"#cite_note-FriedmanEvaluation06-14\" rel=\"external_link\">[14]<\/a><\/sup> Several researchers have, therefore, recommended evaluation studies specially focused on PACSs to assess their impact in clinical practice [4,15].<sup id=\"rdp-ebb-cite_ref-HainsTheImpact12_4-2\" class=\"reference\"><a href=\"#cite_note-HainsTheImpact12-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MansooriPicture12_15-0\" class=\"reference\"><a href=\"#cite_note-MansooriPicture12-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/p><p>It is of paramount significance to investigate the success of the PACS, exploring the factors responsible for the success or failure to determine its worth clinically, based on the direct users of this system.\n<\/p><p>The conceptual basis of this study is focused on this: the impact of a PACS in a regional hospital in Kuwait based on specific criteria. This study is the first of its kind in Kuwait, there being a scarcity of literature in this field.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Research_questions\">Research questions<\/span><\/h3>\n<p>The research questions were specifically as follows: (1) What impact does the PACS have on the clinical practice of radiologists and technologists in the radiology department of Mubarak Al-Kabeer Hospital? (2) Has the use of the PACS proven successful in improving the radiology department\u2019s work performance?\n<\/p><p>This study aimed at evaluating the success of the PACS in clinical practice, in a bid to determine the technology\u2019s merits for radiologists and technologists, including its drawbacks.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Research_settings\">Research settings<\/span><\/h3>\n<p>The universe of this study was Mubarak Al-Kabeer Teaching Hospital, which is one of the five regional hospitals in the State of Kuwait. Table 1 presents the site\u2019s profile. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> Mubarak Al-Kabeer teaching hospital\u2019s profile\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Categories\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hospital beds\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">734\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hospitalized patients\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">21,124\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Physicians\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">559\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Radiologists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">52\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Radiology technologists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">34\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PACS administrators\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Average number of images examined monthly\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">32,787\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>This general hospital is a university-teaching hospital in Kuwait and was chosen because it is always at the forefront of development and advanced medicine. Therefore, to ensure the full advantage of the health information system (HIS), the PACS\u2019s success needed to be verified. The PACS was first introduced in the radiology department of Mubarak Al-Kabeer Hospital in 2004, marking the transition of clinical services from a film-based system to an electronic-based system. The PACS used is an off-the-shelf, Oracle-based HIS (GE Centrisity RIS i 4.2 plus, GE PACS IW 3.7.3.9 SP 3). The PACS currently has 35 workstations, with a server capacity of 64 terabytes. Radiologists use the PACS to view images through the <a href=\"https:\/\/www.limswiki.org\/index.php\/Radiology_information_system\" title=\"Radiology information system\" target=\"_blank\" class=\"wiki-link\" data-key=\"62d0a667f8b27309ea38a46507cfb500\">radiology information system<\/a> (RIS), which they use to report their cases. The reports generated by the RIS are then sent to the PACS, through which final reports can be sent to the HIS. The treating physician needs to submit an access request to see patients\u2019 images on the PACS. In June 2013, the PACS software was upgraded, and currently the system is fully integrated technically with the RIS and the HIS, providing the users with a secure system.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Study_design\">Study design<\/span><\/h3>\n<p>An in-depth study was conducted by using quantitative and qualitative methods. This mixed-method study was based on: (1) survey questionnaires, which were distributed to gather information from radiologists and technologists in the radiology department of Mubarak Al-Kabeer Hospital and (2) semi structured interviews, which were conducted to gather empirical information from the PACS administrators. Ethical approval for the study was obtained from the research department of the Ministry of Health, Kuwait.\n<\/p><p>To gather the responses of radiologists and technologists concerning the use of the PACS in their clinical practice, a validated questionnaire from a previous study was used.<sup id=\"rdp-ebb-cite_ref-Par.C3.A9Toward05_16-0\" class=\"reference\"><a href=\"#cite_note-Par.C3.A9Toward05-16\" rel=\"external_link\">[16]<\/a><\/sup> The questionnaire was translated from French into English through an official translation office in Kuwait. The English version of the questionnaire was pretested with five radiologists and three technologists to ensure the suitability and usability of the questions. Accordingly, a number of amendments were made to the questionnaire. These included excluding questions that were found to be irrelevant to the technologists\u2019 use of the PACS, which comprised items that focused on retrieving, displaying, comparing, and manipulating images, including confidence level. In addition, a seven-point Likert scale was changed to five points to make it easier and more familiar for the respondents.\n<\/p><p>In this study, evaluating the PACS\u2019s success was based on an integrated multidimensional model, which was constructed from the model primarily developed by DeLone and McLean<sup id=\"rdp-ebb-cite_ref-DeLoneInformation92_17-0\" class=\"reference\"><a href=\"#cite_note-DeLoneInformation92-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeLoneTheDeLone03_18-0\" class=\"reference\"><a href=\"#cite_note-DeLoneTheDeLone03-18\" rel=\"external_link\">[18]<\/a><\/sup>, later developed with two constructs added to the model, namely system continuance intention and confirmation of expectations.<sup id=\"rdp-ebb-cite_ref-Par.C3.A9Toward05_16-1\" class=\"reference\"><a href=\"#cite_note-Par.C3.A9Toward05-16\" rel=\"external_link\">[16]<\/a><\/sup> (See Figure 1.)\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Buabbas_JMIRMedInfo2016_4-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"265ea1b0fdaf92723537125101125d81\"><img alt=\"Fig1 Buabbas JMIRMedInfo2016 4-2.png\" src=\"https:\/\/www.limswiki.org\/images\/8\/80\/Fig1_Buabbas_JMIRMedInfo2016_4-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> An integrated model of picture archiving and communication system (PACS) success<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The questionnaire comprised seven sections (Textbox 1) for assessing the users\u2019 perspectives on eight interrelated dimensions of the PACS success model. These included: (1) perceived system quality; (2) perceived information quality; (3) perceived service quality; (4) system usage; (5) user satisfaction; (6) perceived net benefits; (7) system continuance intention; and (8) confirmed expectations. The questionnaire was distributed to all radiologists and radiology technologists who had used the PACS in their clinical practice for the last two years.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"1\"><b>Textbox 1.<\/b> Sections of the questionnaire\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Section 1: Quality of PACS<br \/>\n<p>Ease of access and use<br \/> <br \/>* Diversity of functionalities offered by the PACS<br \/>* Reliability of the hardware and software<br \/>* PACS integration and compatibility with the RIS and the HIS<br \/>* Security of the PACS\n<\/p>\n<\/td><\/tr><\/table>\n<\/td><\/tr><\/table>\n<p>The data gathered through the questionnaire were complemented by conducting semi-structured interviews with PACS administrators to gain an understanding of the prevailing clinical environment, which entails them communicating with radiologists, doctors, and technologists, including providing information technology services and support.<sup id=\"rdp-ebb-cite_ref-JoshiPACS14_19-0\" class=\"reference\"><a href=\"#cite_note-JoshiPACS14-19\" rel=\"external_link\">[19]<\/a><\/sup> Their experience further enriched the information gathered and the study\u2019s purpose.\n<\/p><p>The focus of the interviews was primarily similar to that of the questionnaire: to gain a deeper insight into the response patterns of the respondents. The interviews were conducted with the radiology technologists, who are responsible for administering the PACS and overseeing the RIS operations in the radiology department.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Statistical_analysis\">Statistical analysis<\/span><\/h3>\n<p>Data management, analysis, and graphical presentation were carried out using the software Statistical Package for the Social Sciences (SPSS), version 22.0. The questionnaire was evaluated for internal consistency and reliability, and Cronbach alpha values were estimated for major perspectives by combining the Likert scale items for specific aspects, including quality, information, images, technical support and usage, user satisfaction, and overall opinion on the PACS. The descriptive statistics analysis generated frequencies and percentages for all the five-point Likert scale items (1 as lowest or strongly disagree and 5 as highest or strongly agree) in the questionnaire. The Likert scale data were also analyzed to find average values for overall responses and to compare the mean (\u00b1standard deviation, SD) between radiologists and technologists using <i>t<\/i> tests or nonparametric Mann-Whitney tests. The quantitative or continuous variables, age, duration of use (h), and minutes saved every day were first ascertained for normal distribution, applying the Kolmogorov-Smirnov test and were presented as mean \u00b1 SD and range for normally distributed variables and as median, range and interquartile (IQ) for skewed data. The chi-square or Fisher exact test was applied to find any association or significant difference between categorical variables. The Spearman correlation coefficient (rho) was used to find any correlations among the number of hours worked, the use of the PACS, and the minutes saved in daily practice. The two-tailed probability value <i>P<\/i><.05 was considered statistically significant.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Questionnaires\">Questionnaires<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Respondent_demographics\">Respondent demographics<\/span><\/h4>\n<p>The study\u2019s overall response rate was 70%: 75% of the radiologists and 62% of the technologists of the radiology department. The study had 60 respondents: 39 radiologists (mean age = 36\u00b17.5 SD) and 21 technologists (mean age = 28\u00b110 SD). The respondents\u2019 ages varied between 20 and 60 years, with the majority (85%; 51 of 60) aged younger than 40 years. The respondents\u2019 average self-rated level of familiarity with computers was 4.8 \u00b1 1.34 (mean \u00b1 SD) on a scale of 1-7, and 41% (24 of 59) of the respondents had earlier experience with PACSs before working at this radiology department.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Evaluation_of_Different_Perspectives_on_the_PACS\">Evaluation of Different Perspectives on the PACS<\/span><\/h4>\n<p>The overall responses on different perspectives were analyzed, and composite reliability and coefficients (Cronbach alpha) were computed and presented in Table 2, along with mean and range for each perspective. The Cronbach alpha values ranged between .73 and .96, except for one as shown in in Table 2. System quality, images produced, and services all had high (>.9) Cronbach alpha values.\n<\/p><p>The overall perspectives of users have been presented on the following aspects:\n<\/p><p><b>System quality<\/b>\n<\/p><p>Almost three-quarters (75%; 44 of 59) of the respondents rated the PACS positively and as user-friendly, with a mean of 3.28 (Table 2). Comparatively fewer (64%; 38 of 59) respondents mentioned some drawbacks of the system, such as it being temporarily out-of-service or not working, having numerous bugs, causing wait times at the workstations, and exhibiting reduced screen quality which slowed down PACS use. The majority (81%; 48 of 59) agreed that the PACS had improved the quality of services at the radiology department (mean=4.01). However, some suggestions were provided by respondents (mean=3.57) with regard to the system\u2019s improvement, including the provision of more options and investment in upgrading the visualization equipment (PC monitors).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 2.<\/b> PACS users and their responses<br \/> <br \/><sup>a<\/sup>Cronbach Alpha: Measure of Internal Consistency Reliability<br \/><sup>b<\/sup>Mean values are based on a five-point Likert scale, with 1 being the lowest and 5 being the highest.\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">User perspectives of the PACS\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of items\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Alpha<sup>a<\/sup>\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Mean<sup>b<\/sup>\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Range\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Quality<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Encouraging features\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.906\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.284\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.567-4.033\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Non-encouraging features\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.767\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.50-3.400\n<\/td><\/tr>\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Information<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Produce better information\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.888\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.754\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.650-4.000\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Images<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Quality of images produced\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.910\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.272\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.183-4.333\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Compared to traditional films\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.855\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.333\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.100-4.483\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Confidence in image quality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.875\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.205\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.154-4.256\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data adequacy\u2014access to patient data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.808\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.558\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.500-3.617\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Technical support<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reliable, prompt services\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.961\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.598\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.483-3.683\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Use of the PACS and satisfaction<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Frequency of PACS use\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.638\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.573\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.583-4.000\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">User satisfaction\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.887\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.650\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.533-3.717\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Future use of the PACS<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Expectations, and continuance of use\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.734\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.394\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.233-3.483\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Overall opinion and impact of the PACS<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Improved quality and services (benefits)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.919\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.008\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.169-4.390\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><b>Information quality<\/b>\n<\/p><p>In all, 90% (53 of 59) agreed that the PACS produced better and higher-quality information (mean=3.75) that was accurate, updated, relevant, and timely. The system also provided complete patient information, including adequate access to patients\u2019 historical data (mean=3.56).\n<\/p><p><b>Image quality<\/b>\n<\/p><p>The PACS users were extremely satisfied with regard to the quality of the images produced, ease of understanding, and relevance (mean=4.27). They found that the PACS produced much better images compared to traditional film (mean=4.33).\n<\/p><p><b>Technical support and services<\/b>\n<\/p><p>The PACS users were quite satisfied with technical support (mean=3.60) and the reliability, promptness, and dependability of services.\n<\/p><p><b>Use of PACS and satisfaction<\/b>\n<\/p><p>In all, 50% (30 of 60) of the respondents mentioned using the PACS for more than 30 hours per week (Figure 2), although a significant difference was found regarding the duration of PACS use (hours\/week) between radiologists and technologists (P<.001). A high level of user satisfaction was shown with regard to their experience in using the PACS (mean=3.65). The usage of various tools, including changing display format, retrieving, and \u201csplit screening\u201d to compare images, was found to be quite satisfactory (mean=3.57), especially among radiologists.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Buabbas_JMIRMedInfo2016_4-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"052c09ea02fead26a51ae3213027da17\"><img alt=\"Fig2 Buabbas JMIRMedInfo2016 4-2.png\" src=\"https:\/\/www.limswiki.org\/images\/8\/89\/Fig2_Buabbas_JMIRMedInfo2016_4-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> Respondents\u2019 picture archiving and communication system (PACS) use per week<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><b>Future use and expectations on PACS<\/b>\n<\/p><p>In all, 83.9% (mean=3.39) of PACS users mentioned their expectations better than what they expected originally and showed intention to continue using PACS.\n<\/p><p><b>Overall opinions and impact of PACS<\/b>\n<\/p><p>Based on 21 different statements, 93% (56 of 60) of the PACS users showed consensus on various aspects of the system\u2019s benefits and effectiveness (mean=4.01), and the mean was significantly higher for technologists as compared with radiologists (4.22 vs 3.89). Furthermore, the results showed that 80% (48 of 60) of the PACS users reported saving more than 30 minutes of their practice time each day, whereas 38% (23 of 60) mentioned saving more than an hour each day.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Radiologists_versus_technologists\">Radiologists versus technologists<\/span><\/h4>\n<p>Table 3 summarizes the comparison between radiologists\u2019 and technologists\u2019 responses with regard to their perspectives concerning the PACS. The mean values were significantly higher for the technologists as compared with the radiologists, especially concerning quality, information, patient data, technical support, and overall opinion on impact of the PACS (P<.05). Both professionals showed the highest level of satisfaction (mean >4) with regard to image produced, and their overall opinions on PACS demonstrated improved quality and services (radiologist 3.9 and technologists 4.2).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><b>Table 3.<\/b> Radiologists\u2019 and technologists\u2019 responses<br \/> <br \/><sup>a<\/sup>Mean values are based on a five-point Likert scale, with 1 being the lowest and 5 being the highest.<br \/><sup>b<\/sup>Technologists were not asked this question, as the decision on image quality lies on radiologists.\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\" rowspan=\"2\">User perspectives of the PACS\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Radiologists (n=39)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\">Technologists (n=21)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" rowspan=\"2\"><i>P<\/i> value\n<\/th><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Mean<sup>a<\/sup>\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Mean<sup>a<\/sup>\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">SD\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Quality<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Encouraging features\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.109\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.559\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.733\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.528\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.006\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Non-encouraging features\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.070\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.693\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.333\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.563\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.244\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Information<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Produce better information\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.539\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.830\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.155\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.886\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.007\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Images<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Quality of images produced\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.188\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.692\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.429\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.598\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.186\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Compared to traditional films\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.436\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.622\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.143\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.705\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.083\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Confidence in image quality<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.205\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.704\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">---\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">---\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">---\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data adequacy\u2014access to patient data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.295\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.074\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.048\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.879\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.005\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Technical support<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reliable, prompt services\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.396\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.080\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.973\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.600\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.029\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Use of the PACS and satisfaction<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Frequency of PACS use\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.585\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.760\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.552\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.819\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.963\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">User satisfaction\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.556\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.863\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.825\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.611\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.144\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Future use of the PACS<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Expectations, and continuance of use\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.282\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.867\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.603\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.629\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.140\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Overall opinion and impact of the PACS<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Improved quality and services (benefits)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.892\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.623\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.218\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.427\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.050\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In total, 49% (19 of 39) of the radiologists mentioned saving more than 60 minutes every day, as compared to 19% (4 of 21) of the technologists (P=.048) (Figure 3).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Buabbas_JMIRMedInfo2016_4-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"99c7669df70b05aa1cb9c756c0ecf2e5\"><img alt=\"Fig3 Buabbas JMIRMedInfo2016 4-2.png\" src=\"https:\/\/www.limswiki.org\/images\/d\/dc\/Fig3_Buabbas_JMIRMedInfo2016_4-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3<\/b> Respondents\u2019 minutes saved per day<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>During use of the PACS, both professional groups reported significant savings in the working time for different modalities, though with much variation (the Kolmogorov\u2013Smirnov tests showed a skewed distribution); the median and interquartile have been presented in Figure 4 as a box-plot. The maximum number of minutes saved was 52 minutes (median time) by radiologists in magnetic resonance imaging and 50 minutes by technologists in radiography.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Buabbas_JMIRMedInfo2016_4-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"7ecc49a9f6a2e0f4c15cd23974b5604d\"><img alt=\"Fig4 Buabbas JMIRMedInfo2016 4-2.png\" src=\"https:\/\/www.limswiki.org\/images\/3\/37\/Fig4_Buabbas_JMIRMedInfo2016_4-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 4<\/b> Average minutes (median with interquartile range) saved per day by picture archiving and communication system (PACS) users in different modalities<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>A significant positive correlation was observed between the number of hours using the PACS and the minutes saved in daily practice since the introduction of the PACS (r=0.27, P=.037).\n<\/p><p>The level of prior familiarity with computers was found to be similar between the radiologists (4.84\u00b11.34 SD) and the technologists (4.71\u00b11.35 SD) and did not make any significant difference either in the average duration (hours\/week) of working with the PACS or the time saved (minutes\/day) during practice.\n<\/p><p>The results of the open-ended questions showed that 24% (9 of 38) of the radiologists and 33% (7 of 21) of the technologists stated that storing, retrieving, and comparing images were the most positive elements associated with the use of the PACS. By contrast, 33% (13 of 39) of the radiologists and 43% (9 of 21) of the technologists stated that frequent glitches were the most negative element associated with the PACS.\n<\/p><p>Overall, the study\u2019s findings revealed that both the radiologists and the technologists perceived the adoption of the PACS positively. The mean scores were mostly above 3 or 4 on a scale of 1-5. The mean scores for image quality and information produced were 4.3 and 3.8, respectively. The users seemed quite satisfied with the services and technical support, with a mean score of 3.6 and showed satisfaction in working with the PACS (mean=3.65). The PACS users clearly mentioned improved services and quality since the system came into practice, with a mean score of 4.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Interviews\">Interviews<\/span><\/h4>\n<p>The opinions of the PACS administrators were obtained by using the interview method, for which a series of semi-structured questions on specific themes (Textbox 1) provided the basis for soliciting information.\n<\/p><p>At the time of the interviews, Mubarak Al-Kabeer Hospital had five PACS administrators. For scheduling the interviews, requisite permission was taken from the head of the radiology department, and interview sessions were arranged with the staff during their respective work breaks, over a five-day period. Each interview session lasted approximately 50 minutes. The interviews were transcribed, and the responses were coded and analyzed using thematic analysis.\n<\/p><p>The interview results showed that all the interviewees had a BSc degree in radiological sciences, with their ages ranging between 25 and 35 years, and each having work experience of 2-5 years in PACS administration. Of the interviewees, only three had undertaken an introductory training program abroad on PACS use and management.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Perceived_system_quality\">Perceived system quality<\/span><\/h4>\n<p>The interview responses confirmed that the PACS provided easy access to authorized users, each with a user identification (ID) number and password, thereby providing a secured workspace depending on the user\u2019s position. For instance, a radiology technologist\u2019s access is limited to only viewing the reporting screen, with no authorization to change or manipulate it, thus preserving the data, with no hacking or security problems ever encountered or reported.\n<\/p><p>The interviewees unanimously agreed that the PACS was user friendly and hassle free in its functionality. In one of the interviewee\u2019s words, \u201cWe haven\u2019t experienced any complaints from radiologists regarding the clarity of the PACS\u2019s features, or any difficulties in moving between its functions,\u201d further adding that training in the PACS should be a prerequisite before its use.\n<\/p><p>The participants also endorsed the reliability and consistency of the existing hardware, including computer systems, networks, and printers, with the software used. The interviews further revealed that the PACS was fully integrated and compatible with the RIS and the HIS, although the workflow did not follow the planned process, as Figure 5 demonstrates. According to one interviewee, \u201cThe real mistakes are not coming from the PACS but from humans, so they\u2019re human errors.\u201d The interview responses also highlighted that the problems associated with PACS integration and compatibility with the RIS and the HIS were the result of disorganized workflow, as shown in Figure 5.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Buabbas_JMIRMedInfo2016_4-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"e558ffee500d83aff34e42dfe6242c33\"><img alt=\"Fig5 Buabbas JMIRMedInfo2016 4-2.png\" src=\"https:\/\/www.limswiki.org\/images\/a\/aa\/Fig5_Buabbas_JMIRMedInfo2016_4-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 5<\/b> Workstations where electronic registrations of patients failed through the health information system (HIS) and radiology information system (RIS)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Figure 5 (above) illustrates the workstations where electronic registration of patients through the HIS and the RIS failed due to receptionist errors such as: (1) no data entry into HIS, manual registration in the RIS; (2) failure of communication between the HIS and the RIS, manual registration in the RIS; (3) and incorrect registration at reception, manual registration in the RIS (Figure 6).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Buabbas_JMIRMedInfo2016_4-2.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"400e1d22d5c15b85d313774a0df9717f\"><img alt=\"Fig6 Buabbas JMIRMedInfo2016 4-2.png\" src=\"https:\/\/www.limswiki.org\/images\/8\/8b\/Fig6_Buabbas_JMIRMedInfo2016_4-2.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 6<\/b> Manual registration of patients through the health information system (HIS) and radiology information system (RIS)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The manual registration at these three workstations resulted in: (1) a lack of direct access to patients\u2019 imaging results through the HIS; (2) the creation of multiple PACS numbers for the same patient, making it difficult to retrieve previous reports for comparison, as well as the loss of patient data; and (3) delayed patient case management due to a failure in the rapid delivery of results.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Perceived_information_quality\">Perceived information quality<\/span><\/h4>\n<p>The interviewees agreed that the PACS provided a standard format for the acquisition of accurate and complete information, together with images, concerning the patients\u2019 medical cases, including their name, age, gender, national identification number, medical record number, and medical history. The lapses that occurred in the recorded information were attributed to the registration staff of the diagnostic radiology department because of their noncompliance to instructions, which resulted in incomplete data records of patients at the time of registration.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Perceived_image_quality\">Perceived image quality<\/span><\/h4>\n<p>The interview responses indicated that one of the main roles of PACS administrators was to ensure that the images were transferred and displayed with clarity to facilitate studying and reporting. The participants further confirmed that \u201cWe experience hangs in the images in PACS, but at an acceptable rate\u201d and no complaints were mentioned concerning image manipulation and management.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Perceived_technical_support_services\">Perceived technical support services<\/span><\/h4>\n<p>As the interviewees mentioned, the main IT support is delivered through the company that sold the PACS. This usually happens when the PACS administrators face a technical problem that can only be solved through the main IT support at the company. Thus, the PACS administrators asked to have some power to authorize them to solve the technical issues within the radiology department. One of the interviewees stated: \u201c\u2026even when we want to connect a new printer to the PACS, we have to call the main IT support to perform this function for us.\u201d However, all of the interviewees complimented the IT support services at the company for their prompt responses to any technical issues.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Impact_of_PACS_on_clinical_practice\">Impact of PACS on clinical practice<\/span><\/h4>\n<p>PACS has an impact on the clinical practice of radiologists and technologists, as shown in the interviews' results.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Perceived_net_benefits_of_the_PACS\">Perceived net benefits of the PACS<\/span><\/h4>\n<p>From the interviews, it was easy to see that the PACS has increased users\u2019 productivity in comparison to the traditional filming system by minimizing their effort and time. In addition, the retaking of images is not required, as the PACS facilitates image storage and retrieval faster and over a longer period. \u201cWe are happy with the PACS\u2019s benefits,\u201d reported one interviewee, although the system has slowed in speed due to the huge number of cases, with the intervening procedures passing through several modalities, such as computed tomography and magnetic resonance imaging. There is also the possibility of missed images, especially concerning unknown IDs, although these could be traced using the patient\u2019s civil ID, the patient\u2019s PACS ID, or the excision ID of images.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"User_satisfaction_with_the_PACS\">User satisfaction with the PACS<\/span><\/h4>\n<p>All the interviewees were apparently satisfied with the PACS; however, the technology-associated problems need to be addressed to optimize the system\u2019s versatility and performance.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Opinions_on_the_PACS\">Opinions on the PACS<\/span><\/h4>\n<p>Overall, from the interviews, the responses revealed that as long as the image is electronically collected, stored, and communicated to another system successfully, the productively of work will be increased, diagnosis will be precisely performed, the patient will be treated accurately and quickly, and health services will be improved.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Expectations_of_the_current_PACS_and_future_trends\">Expectations of the current PACS and future trends<\/span><\/h4>\n<p>The interviewees expressed satisfaction in using the PACS system but also highlighted the need for resolving the current problems, as well as to keep abreast of the latest advances in PACS operations, to meet the growing demands of the Kuwaiti health system. The emerging requirements for potential trends in the future concern the areas of: (1) teleradiology services (for radiologists to use the PACS anywhere and anytime); (2) a mobile image viewer for faster accessibility to images; (3) speech recognition functions; (4) computer-assisted diagnosis (CAD); (5) advanced training; and (6) recruiting health informatics graduates to support the PACS administrators.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>In general, the study\u2019s findings revealed that the PACS has had a productive impact on the staff\u2019s clinical practice. Despite some of the technical limitations of the infrastructure, most of the respondents rated the system positively and as user-friendly. The findings showed that the technologists were more satisfied than the radiologists were with using the PACS. Interestingly, there was a significant relationship between the perceived benefits of the PACS and the willingness of users to continue using it. It was also noteworthy that the problems associated with the PACS\u2019s integration with the RIS were the result of disorganized workflow.\n<\/p><p>The results of the study revealed that the users\u2019 demographic data, including computer experience, had no influence on their response patterns, being insignificant determinants of their predilection or preference for the PACS in enhancing their work efficiency. These findings were consistent with the study\u2019s results on PACS acceptance<sup id=\"rdp-ebb-cite_ref-AldosariUser12_7-1\" class=\"reference\"><a href=\"#cite_note-AldosariUser12-7\" rel=\"external_link\">[7]<\/a><\/sup>, but contradicted with the results of earlier studies that reported the significant influence of age and gender on users\u2019 choices concerning information technology, such as computer use patterns<sup id=\"rdp-ebb-cite_ref-WardTheAttitudes08_20-0\" class=\"reference\"><a href=\"#cite_note-WardTheAttitudes08-20\" rel=\"external_link\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AdelekeICT15_21-0\" class=\"reference\"><a href=\"#cite_note-AdelekeICT15-21\" rel=\"external_link\">[21]<\/a><\/sup>, particularly to adopting PACS.<sup id=\"rdp-ebb-cite_ref-DuyckMonitoring10_3-1\" class=\"reference\"><a href=\"#cite_note-DuyckMonitoring10-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BryanEval99_8-1\" class=\"reference\"><a href=\"#cite_note-BryanEval99-8\" rel=\"external_link\">[8]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Perceptions_of_PACS_quality.2C_information.2C_images.2C_and_services\">Perceptions of PACS quality, information, images, and services<\/span><\/h3>\n<p>The study further revealed that both the radiologists and the technologists were satisfied with the quality of information and images produced and had positive views regarding the use of this technology. The PACS offered the users with the requisite information on a medical case and facilitated the accomplishment of several functions with efficiency and ease in producing high-quality images with precision and clarity. This positive relationship found between users' satisfaction and quality of information and images produced by PACS was consistent with the findings of previous studies.<sup id=\"rdp-ebb-cite_ref-TzengASocio13_1-1\" class=\"reference\"><a href=\"#cite_note-TzengASocio13-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SicotteVirtual10_22-0\" class=\"reference\"><a href=\"#cite_note-SicotteVirtual10-22\" rel=\"external_link\">[22]<\/a><\/sup> The results of the interviews further complemented these findings, with no mention of lost images posing a major problem, due to successful image retrieval by PACS administrators.\n<\/p><p>The study found that the technologists were more satisfied than the radiologists concerning their current PACS use, attributing their satisfaction to two reasons, which had been confirmed in previous studies<sup id=\"rdp-ebb-cite_ref-VestImage15_2-1\" class=\"reference\"><a href=\"#cite_note-VestImage15-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TanPicture10_6-1\" class=\"reference\"><a href=\"#cite_note-TanPicture10-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FolioQuantitative15_23-0\" class=\"reference\"><a href=\"#cite_note-FolioQuantitative15-23\" rel=\"external_link\">[23]<\/a><\/sup>: the technologists achieved their core objectives of using the PACS, including image access, storage, and retrieval; and (2) the radiologists looked beyond these features for additional facilities and functions, such as the PACS being packaged with CAD, teleradiology, or speech recognition functions. As the radiologists had been using the PACS far longer than the technologists had, their understanding and familiarity with the PACS appeared to be relatively higher.\n<\/p><p>Concerning the quality of the services offered to support PACS technically, the findings showed that both users were satisfied with the technical support provided with regard to the promptness, reliability, and dependability of the services. However, the results of the interviews revealed that the radiologists and the technologists encountered organizational and infrastructure deficiencies. On the technical level, there was frequent breakdown of the system during rush hours; and on organizational level, there was negligence of some receptionists in recording patients\u2019 information from the RIS to the HIS. Interestingly, the respondents still showed satisfaction in confirming the benefits of the PACS over conventional radiology despite some deficiencies, as reportedly addressed in previous studies.<sup id=\"rdp-ebb-cite_ref-DuyckMonitoring10_3-2\" class=\"reference\"><a href=\"#cite_note-DuyckMonitoring10-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SicotteVirtual10_22-1\" class=\"reference\"><a href=\"#cite_note-SicotteVirtual10-22\" rel=\"external_link\">[22]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Perceptions_of_the_PACS.E2.80.99s_impact.2C_including_net_benefits_and_user_satisfaction\">Perceptions of the PACS\u2019s impact, including net benefits and user satisfaction<\/span><\/h3>\n<p>Regarding the PACS\u2019s net benefits, the findings demonstrated that both the radiologists and the technologists had used the PACS to enhance their work productivity with ease due to the swift storage, retrieval, and transfer of images along with reports. These findings were consistent with those reported in previous evaluative studies on the impact of PACS<sup id=\"rdp-ebb-cite_ref-TanPicture10_6-2\" class=\"reference\"><a href=\"#cite_note-TanPicture10-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SevensterImproved15_24-0\" class=\"reference\"><a href=\"#cite_note-SevensterImproved15-24\" rel=\"external_link\">[24]<\/a><\/sup>, confirming that work productivity in regard to the given effort, time, and accuracy of reporting has obviously been improved. Furthermore, the PACS\u2019s benefits were found to have direct implications for user satisfaction, affecting their continued use of the PACS in the future.<sup id=\"rdp-ebb-cite_ref-Par.C3.A9Toward05_16-2\" class=\"reference\"><a href=\"#cite_note-Par.C3.A9Toward05-16\" rel=\"external_link\">[16]<\/a><\/sup> These previous studies concluded that the more the users agreed with a PACS\u2019s effectiveness in their work, the more they were satisfied and willing to continue using it. The findings of the interviews further confirmed that both types of users benefited from the PACS\u2019s advantages, expressing their readiness toward the technology\u2019s continuous use while looking ahead for additional functions, without deficiencies, which coincided with other studies.<sup id=\"rdp-ebb-cite_ref-HsiaoCritical16_25-0\" class=\"reference\"><a href=\"#cite_note-HsiaoCritical16-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiuExploring15_26-0\" class=\"reference\"><a href=\"#cite_note-LiuExploring15-26\" rel=\"external_link\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FloydStudy15_27-0\" class=\"reference\"><a href=\"#cite_note-FloydStudy15-27\" rel=\"external_link\">[27]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Limitations\">Limitations<\/span><\/h3>\n<p>(1) This study was limited to radiologists and radiology technologists and did not involve other health care providers who are responsible for receiving patients\u2019 reports and images. Hence, there is a need for further research that would substantiate the study\u2019s findings by involving other stakeholders using the PACS facility, for the purpose of comparing research outcomes and enhancing the study\u2019s value. (2) The study also did not include socioeconomic and cultural factors, which are significant predictors of IT adoption in the Arab world<sup id=\"rdp-ebb-cite_ref-AldosariUser12_7-2\" class=\"reference\"><a href=\"#cite_note-AldosariUser12-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RayClin17_28-0\" class=\"reference\"><a href=\"#cite_note-RayClin17-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-StraubTransfer01_29-0\" class=\"reference\"><a href=\"#cite_note-StraubTransfer01-29\" rel=\"external_link\">[29]<\/a><\/sup> in comparison to Western countries. However, the respondents\u2019 willingness to use the PACS was a positive indicator of the technology\u2019s versatility, efficiency, and continuous use. (3) As the study was confined to one general hospital in Kuwait, there is a definitive need for future studies to enhance the study\u2019s scope by including other hospitals where PACSs are being used, for comparative purposes. (4) The study used specific criteria in evaluating IS success; hence, there is a need for using different models and tools for exploring and assessing PACSs and RISs from different dimensions.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h3>\n<p>Evaluating the applications of imaging informatics, such as PACSs, in hospitals is crucial to ensure the successful implementation of the applications, to identify the systems\u2019 strengths and weaknesses during operation, and to provide the opportunity for further improvements, strengthening the positive elements and minimizing drawbacks.\nThe evaluation of the existing PACS at Mubarak Al-Kabeer teaching hospital led to the successful assessment of the technology\u2019s implications, based on which the study\u2019s conclusions are summarized: (1) the PACS exhibited a positive impact on the radiologists and the technologists in the diagnostic radiology department, significantly enhancing their work efficiency and productivity. Therefore, the impact of the technology was particularly visible in the context of its ability to store and retrieve images quickly, enabling the users to accomplish their tasks swiftly. In addition, the system facilitated the addition of an image to a report, expediting communication with another location with a keystroke; (2) the main concern reported by all the users was the frequent breakdown during rush hours at busy workstations, due to infrastructure deficiency; (3) both the technologists and the radiologists indicated the need for a more-advanced PACS in response to the growing demand of teleradiology, mobile image viewing, and voice recognition features; and (4) evaluating PACS\u2019s success is not confined to the technology itself but also concerns organizational and human factors that could limit the full integration with HIS.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Recommendations\">Recommendations<\/span><\/h3>\n<p>To improve the work on the current PACS and overcome the deficiencies, the following recommendations could be considered at Mubarak Al-Kabeer general hospital: (1) Enhance the capacity of existing servers to accommodate the huge amount of data generated from the massive inflow of patients. (2) Develop an internal policy to facilitate the coordination with the hospital management for organizing hospital workflow with efficiency. This policy should be followed carefully by the department staff for achieving the full benefits of the PACS\u2019s integration with the HIS and the RIS. (3) Offer advanced training courses for fully using the PACS\u2019s functions. (4) Keep a forward-looking eye on future trends of PACS, including teleradiology services, mobile images viewing, speech recognition, and CAD. (5) Hire health informatics specialists for providing the requisite administrative support on account of their knowledge in the field.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgments\">Acknowledgments<\/span><\/h2>\n<p>Special thanks go to all the staff at the radiology department of Mubarak Al-Kabeer Teaching Hospital in Kuwait for their efforts and contributions in making this study successful.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h2>\n<p>None declared.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-TzengASocio13-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TzengASocio13_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-TzengASocio13_1-1\" rel=\"external_link\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tzeng, W.S.; Kuo, K.M.; Lin, H.W.; Chen, T.Y. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849362\" target=\"_blank\">\"A socio-technical assessment of the success of Picture Archiving and Communication Systems: The radiology technologist's perspective\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>13<\/b>: 109. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-13-109\" target=\"_blank\">10.1186\/1472-6947-13-109<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3849362\/\" target=\"_blank\">PMC3849362<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24053458\" target=\"_blank\">24053458<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849362\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849362<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+socio-technical+assessment+of+the+success+of+Picture+Archiving+and+Communication+Systems%3A+The+radiology+technologist%27s+perspective&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Tzeng%2C+W.S.%3B+Kuo%2C+K.M.%3B+Lin%2C+H.W.%3B+Chen%2C+T.Y.&rft.au=Tzeng%2C+W.S.%3B+Kuo%2C+K.M.%3B+Lin%2C+H.W.%3B+Chen%2C+T.Y.&rft.date=2013&rft.volume=13&rft.pages=109&rft_id=info:doi\/10.1186%2F1472-6947-13-109&rft_id=info:pmc\/PMC3849362&rft_id=info:pmid\/24053458&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3849362&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VestImage15-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VestImage15_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-VestImage15_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vest, J.R.; Jung, H.Y.; Ostrovsky, A. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4730956\" target=\"_blank\">\"Image Sharing Technologies and Reduction of Imaging Utilization: A Systematic Review and Meta-analysis\"<\/a>. <i>Journal of the American College of Radiology<\/i> <b>12<\/b> (12 Pt B): 1371-1379.e3. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jacr.2015.09.014\" target=\"_blank\">10.1016\/j.jacr.2015.09.014<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4730956\/\" target=\"_blank\">PMC4730956<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26614882\" target=\"_blank\">26614882<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4730956\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4730956<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Image+Sharing+Technologies+and+Reduction+of+Imaging+Utilization%3A+A+Systematic+Review+and+Meta-analysis&rft.jtitle=Journal+of+the+American+College+of+Radiology&rft.aulast=Vest%2C+J.R.%3B+Jung%2C+H.Y.%3B+Ostrovsky%2C+A.+et+al.&rft.au=Vest%2C+J.R.%3B+Jung%2C+H.Y.%3B+Ostrovsky%2C+A.+et+al.&rft.date=2015&rft.volume=12&rft.issue=12+Pt+B&rft.pages=1371-1379.e3&rft_id=info:doi\/10.1016%2Fj.jacr.2015.09.014&rft_id=info:pmc\/PMC4730956&rft_id=info:pmid\/26614882&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4730956&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DuyckMonitoring10-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DuyckMonitoring10_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-DuyckMonitoring10_3-1\" rel=\"external_link\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-DuyckMonitoring10_3-2\" rel=\"external_link\">3.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Duyck, P.; Pynoo, B.; Devolder, P. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3043749\" target=\"_blank\">\"Monitoring the PACS implementation process in a large university hospital: Discrepancies between radiologists and physicians\"<\/a>. <i>Journal of Digital Imaging<\/i> <b>23<\/b> (1): 73\u201380. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-008-9163-7\" target=\"_blank\">10.1007\/s10278-008-9163-7<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3043749\/\" target=\"_blank\">PMC3043749<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18956231\" target=\"_blank\">18956231<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3043749\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3043749<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Monitoring+the+PACS+implementation+process+in+a+large+university+hospital%3A+Discrepancies+between+radiologists+and+physicians&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=Duyck%2C+P.%3B+Pynoo%2C+B.%3B+Devolder%2C+P.+et+al.&rft.au=Duyck%2C+P.%3B+Pynoo%2C+B.%3B+Devolder%2C+P.+et+al.&rft.date=2010&rft.volume=23&rft.issue=1&rft.pages=73%E2%80%9380&rft_id=info:doi\/10.1007%2Fs10278-008-9163-7&rft_id=info:pmc\/PMC3043749&rft_id=info:pmid\/18956231&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3043749&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HainsTheImpact12-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HainsTheImpact12_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-HainsTheImpact12_4-1\" rel=\"external_link\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-HainsTheImpact12_4-2\" rel=\"external_link\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hains, I.M.; Georgiou, A.; Westbrook, J.I. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3384105\" target=\"_blank\">\"The impact of PACS on clinician work practices in the intensive care unit: a systematic review of the literature\"<\/a>. <i>JAMIA<\/i> <b>19<\/b> (4): 506\u201313. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2011-000422\" target=\"_blank\">10.1136\/amiajnl-2011-000422<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3384105\/\" target=\"_blank\">PMC3384105<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22323392\" target=\"_blank\">22323392<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3384105\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3384105<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+impact+of+PACS+on+clinician+work+practices+in+the+intensive+care+unit%3A+a+systematic+review+of+the+literature&rft.jtitle=JAMIA&rft.aulast=Hains%2C+I.M.%3B+Georgiou%2C+A.%3B+Westbrook%2C+J.I.&rft.au=Hains%2C+I.M.%3B+Georgiou%2C+A.%3B+Westbrook%2C+J.I.&rft.date=2012&rft.volume=19&rft.issue=4&rft.pages=506%E2%80%9313&rft_id=info:doi\/10.1136%2Famiajnl-2011-000422&rft_id=info:pmc\/PMC3384105&rft_id=info:pmid\/22323392&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3384105&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BryanTheBenefits99-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BryanTheBenefits99_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bryan, S.; Weatherburn, G.C.; Watkins, J.R.; Buxton, M.J. (1999). \"The benefits of hospital-wide picture archiving and communication systems: A survey of clinical users of radiology services\". <i>The British Journal of Radiology<\/i> <b>72<\/b> (857): 469-78. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1259%2Fbjr.72.857.10505012\" target=\"_blank\">10.1259\/bjr.72.857.10505012<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10505012\" target=\"_blank\">10505012<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+benefits+of+hospital-wide+picture+archiving+and+communication+systems%3A+A+survey+of+clinical+users+of+radiology+services&rft.jtitle=The+British+Journal+of+Radiology&rft.aulast=Bryan%2C+S.%3B+Weatherburn%2C+G.C.%3B+Watkins%2C+J.R.%3B+Buxton%2C+M.J.&rft.au=Bryan%2C+S.%3B+Weatherburn%2C+G.C.%3B+Watkins%2C+J.R.%3B+Buxton%2C+M.J.&rft.date=1999&rft.volume=72&rft.issue=857&rft.pages=469-78&rft_id=info:doi\/10.1259%2Fbjr.72.857.10505012&rft_id=info:pmid\/10505012&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TanPicture10-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TanPicture10_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-TanPicture10_6-1\" rel=\"external_link\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-TanPicture10_6-2\" rel=\"external_link\">6.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tan, S.L.; Lewis, R.A. (2010). \"Picture archiving and communication systems: A multicentre survey of users experience and satisfaction\". <i>European Journal of Radiology<\/i> <b>75<\/b> (3): 406\u201310. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ejrad.2009.05.008\" target=\"_blank\">10.1016\/j.ejrad.2009.05.008<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19523778\" target=\"_blank\">19523778<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Picture+archiving+and+communication+systems%3A+A+multicentre+survey+of+users+experience+and+satisfaction&rft.jtitle=European+Journal+of+Radiology&rft.aulast=Tan%2C+S.L.%3B+Lewis%2C+R.A.&rft.au=Tan%2C+S.L.%3B+Lewis%2C+R.A.&rft.date=2010&rft.volume=75&rft.issue=3&rft.pages=406%E2%80%9310&rft_id=info:doi\/10.1016%2Fj.ejrad.2009.05.008&rft_id=info:pmid\/19523778&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AldosariUser12-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AldosariUser12_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-AldosariUser12_7-1\" rel=\"external_link\">7.1<\/a><\/sup> <sup><a href=\"#cite_ref-AldosariUser12_7-2\" rel=\"external_link\">7.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aldosari, B. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3423046\" target=\"_blank\">\"User acceptance of a picture archiving and communication system (PACS) in a Saudi Arabian hospital radiology department\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>12<\/b>: 44. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-12-44\" target=\"_blank\">10.1186\/1472-6947-12-44<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3423046\/\" target=\"_blank\">PMC3423046<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22640490\" target=\"_blank\">22640490<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3423046\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3423046<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=User+acceptance+of+a+picture+archiving+and+communication+system+%28PACS%29+in+a+Saudi+Arabian+hospital+radiology+department&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Aldosari%2C+B.&rft.au=Aldosari%2C+B.&rft.date=2012&rft.volume=12&rft.pages=44&rft_id=info:doi\/10.1186%2F1472-6947-12-44&rft_id=info:pmc\/PMC3423046&rft_id=info:pmid\/22640490&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3423046&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BryanEval99-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BryanEval99_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-BryanEval99_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bryan, S.; Weatherburn, G.; Buxton, M. et al. (1999). \"Evaluation of a hospital picture archiving and communication system\". <i>Journal of Health Services Research and Policy<\/i> <b>4<\/b> (4): 204-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F135581969900400405\" target=\"_blank\">10.1177\/135581969900400405<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10623035\" target=\"_blank\">10623035<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+of+a+hospital+picture+archiving+and+communication+system&rft.jtitle=Journal+of+Health+Services+Research+and+Policy&rft.aulast=Bryan%2C+S.%3B+Weatherburn%2C+G.%3B+Buxton%2C+M.+et+al.&rft.au=Bryan%2C+S.%3B+Weatherburn%2C+G.%3B+Buxton%2C+M.+et+al.&rft.date=1999&rft.volume=4&rft.issue=4&rft.pages=204-9&rft_id=info:doi\/10.1177%2F135581969900400405&rft_id=info:pmid\/10623035&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoodleyAComp15-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoodleyAComp15_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moodley, I.; Moodley, S. (2015). \"A comparative cost analysis of picture archiving and communications systems (PACS) versus conventional radiology in the private sector\". <i>South African Journal of Radiology<\/i> <b>19<\/b> (1): 634-641. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fsajr.v19i1.634\" target=\"_blank\">10.4102\/sajr.v19i1.634<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comparative+cost+analysis+of+picture+archiving+and+communications+systems+%28PACS%29+versus+conventional+radiology+in+the+private+sector&rft.jtitle=South+African+Journal+of+Radiology&rft.aulast=Moodley%2C+I.%3B+Moodley%2C+S.&rft.au=Moodley%2C+I.%3B+Moodley%2C+S.&rft.date=2015&rft.volume=19&rft.issue=1&rft.pages=634-641&rft_id=info:doi\/10.4102%2Fsajr.v19i1.634&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MackinnonPicture08-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MackinnonPicture08_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mackinnon, A.D.; Billington, R.A.; Adam, E.J. et al. (2008). \"Picture archiving and communication systems lead to sustained improvements in reporting times and productivity: Results of a 5-year audit\". <i>Clinical Radiology<\/i> <b>63<\/b> (7): 796-804. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.crad.2007.12.015\" target=\"_blank\">10.1016\/j.crad.2007.12.015<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18555038\" target=\"_blank\">18555038<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Picture+archiving+and+communication+systems+lead+to+sustained+improvements+in+reporting+times+and+productivity%3A+Results+of+a+5-year+audit&rft.jtitle=Clinical+Radiology&rft.aulast=Mackinnon%2C+A.D.%3B+Billington%2C+R.A.%3B+Adam%2C+E.J.+et+al.&rft.au=Mackinnon%2C+A.D.%3B+Billington%2C+R.A.%3B+Adam%2C+E.J.+et+al.&rft.date=2008&rft.volume=63&rft.issue=7&rft.pages=796-804&rft_id=info:doi\/10.1016%2Fj.crad.2007.12.015&rft_id=info:pmid\/18555038&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VanDeWeteringABalanced06-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VanDeWeteringABalanced06_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">van de Wetering, R.; Batenburg, R.; Versendaal, J. et al. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3045172\" target=\"_blank\">\"A balanced evaluation perspective: Picture archiving and communication system impacts on hospital workflow\"<\/a>. <i>Journal of Digital Imaging<\/i> <b>19<\/b> (Suppl 1): 10. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-006-0628-2\" target=\"_blank\">10.1007\/s10278-006-0628-2<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3045172\/\" target=\"_blank\">PMC3045172<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16763932\" target=\"_blank\">16763932<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3045172\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3045172<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+balanced+evaluation+perspective%3A+Picture+archiving+and+communication+system+impacts+on+hospital+workflow&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=van+de+Wetering%2C+R.%3B+Batenburg%2C+R.%3B+Versendaal%2C+J.+et+al.&rft.au=van+de+Wetering%2C+R.%3B+Batenburg%2C+R.%3B+Versendaal%2C+J.+et+al.&rft.date=2006&rft.volume=19&rft.issue=Suppl+1&rft.pages=10&rft_id=info:doi\/10.1007%2Fs10278-006-0628-2&rft_id=info:pmc\/PMC3045172&rft_id=info:pmid\/16763932&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3045172&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CreswickInnovation11-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CreswickInnovation11_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Creswick, N.; Hains, I.M.; Westbrook, J.I. (2011). \"Innovation in intensive care nursing work practices with PACS\". <i>Studies in Health Technology and Informatics<\/i> <b>169<\/b>: 402\u20136. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2F978-1-60750-806-9-402\" target=\"_blank\">10.3233\/978-1-60750-806-9-402<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21893781\" target=\"_blank\">21893781<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Innovation+in+intensive+care+nursing+work+practices+with+PACS&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Creswick%2C+N.%3B+Hains%2C+I.M.%3B+Westbrook%2C+J.I.&rft.au=Creswick%2C+N.%3B+Hains%2C+I.M.%3B+Westbrook%2C+J.I.&rft.date=2011&rft.volume=169&rft.pages=402%E2%80%936&rft_id=info:doi\/10.3233%2F978-1-60750-806-9-402&rft_id=info:pmid\/21893781&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VanDeWeteringTowards14-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VanDeWeteringTowards14_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">van de Wetering, R.; Batenburg, R. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4026462\" target=\"_blank\">\"Towards a theory of PACS deployment: An integrative PACS maturity framework\"<\/a>. <i>Journal of Digital Imaging<\/i> <b>27<\/b> (3): 337\u201350. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-013-9671-y\" target=\"_blank\">10.1007\/s10278-013-9671-y<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4026462\/\" target=\"_blank\">PMC4026462<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24448919\" target=\"_blank\">24448919<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4026462\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4026462<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+a+theory+of+PACS+deployment%3A+An+integrative+PACS+maturity+framework&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=van+de+Wetering%2C+R.%3B+Batenburg%2C+R.&rft.au=van+de+Wetering%2C+R.%3B+Batenburg%2C+R.&rft.date=2014&rft.volume=27&rft.issue=3&rft.pages=337%E2%80%9350&rft_id=info:doi\/10.1007%2Fs10278-013-9671-y&rft_id=info:pmc\/PMC4026462&rft_id=info:pmid\/24448919&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4026462&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FriedmanEvaluation06-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FriedmanEvaluation06_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-FriedmanEvaluation06_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Friedman, C.P.; Wyatt, J. (2006). <i>Evaluation Methods in Biomedical Informatics<\/i>. Springer-Verlag New York. pp. 386. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F0-387-30677-3\" target=\"_blank\">10.1007\/0-387-30677-3<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780387258898.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Evaluation+Methods+in+Biomedical+Informatics&rft.aulast=Friedman%2C+C.P.%3B+Wyatt%2C+J.&rft.au=Friedman%2C+C.P.%3B+Wyatt%2C+J.&rft.date=2006&rft.pages=pp.%26nbsp%3B386&rft.pub=Springer-Verlag+New+York&rft_id=info:doi\/10.1007%2F0-387-30677-3&rft.isbn=9780387258898&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MansooriPicture12-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MansooriPicture12_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mansoori, B.; Erhard, K.K.; Sunshine, J.L. (2012). \"Picture Archiving and Communication System (PACS) implementation, integration & benefits in an integrated health system\". <i>Academic Radiology<\/i> <b>19<\/b> (2): 229\u201335. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.acra.2011.11.009\" target=\"_blank\">10.1016\/j.acra.2011.11.009<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22212425\" target=\"_blank\">22212425<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Picture+Archiving+and+Communication+System+%28PACS%29+implementation%2C+integration+%26+benefits+in+an+integrated+health+system&rft.jtitle=Academic+Radiology&rft.aulast=Mansoori%2C+B.%3B+Erhard%2C+K.K.%3B+Sunshine%2C+J.L.&rft.au=Mansoori%2C+B.%3B+Erhard%2C+K.K.%3B+Sunshine%2C+J.L.&rft.date=2012&rft.volume=19&rft.issue=2&rft.pages=229%E2%80%9335&rft_id=info:doi\/10.1016%2Fj.acra.2011.11.009&rft_id=info:pmid\/22212425&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Par.C3.A9Toward05-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Par.C3.A9Toward05_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-Par.C3.A9Toward05_16-1\" rel=\"external_link\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-Par.C3.A9Toward05_16-2\" rel=\"external_link\">16.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Par\u00e9, G.; Lepanto, L.; Aubry, D.; Sicotte, C. (2005). \"Toward a multidimensional assessment of picture archiving and communication system success\". <i>International Journal of Technology Assessment in Health Care<\/i> <b>21<\/b> (4): 471\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1017%2FS0266462305050658\" target=\"_blank\">10.1017\/S0266462305050658<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16262970\" target=\"_blank\">16262970<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+multidimensional+assessment+of+picture+archiving+and+communication+system+success&rft.jtitle=International+Journal+of+Technology+Assessment+in+Health+Care&rft.aulast=Par%C3%A9%2C+G.%3B+Lepanto%2C+L.%3B+Aubry%2C+D.%3B+Sicotte%2C+C.&rft.au=Par%C3%A9%2C+G.%3B+Lepanto%2C+L.%3B+Aubry%2C+D.%3B+Sicotte%2C+C.&rft.date=2005&rft.volume=21&rft.issue=4&rft.pages=471%E2%80%939&rft_id=info:doi\/10.1017%2FS0266462305050658&rft_id=info:pmid\/16262970&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeLoneInformation92-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeLoneInformation92_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">DeLone, W.H.; McLean, E.R. (1992). \"Information Systems Success: The Quest for the Dependent Variable\". <i>Information Systems Research<\/i> <b>3<\/b> (1): 60\u201395. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Fisre.3.1.60\" target=\"_blank\">10.1287\/isre.3.1.60<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+Systems+Success%3A+The+Quest+for+the+Dependent+Variable&rft.jtitle=Information+Systems+Research&rft.aulast=DeLone%2C+W.H.%3B+McLean%2C+E.R.&rft.au=DeLone%2C+W.H.%3B+McLean%2C+E.R.&rft.date=1992&rft.volume=3&rft.issue=1&rft.pages=60%E2%80%9395&rft_id=info:doi\/10.1287%2Fisre.3.1.60&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeLoneTheDeLone03-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeLoneTheDeLone03_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">DeLone, W.H.; McLean, E.R. (2003). \"The DeLone and McLean Model of Information Systems Success: A Ten-Year Update\". <i>Journal of Management Information Systems<\/i> <b>19<\/b> (4): 9\u201330. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F07421222.2003.11045748\" target=\"_blank\">10.1080\/07421222.2003.11045748<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+DeLone+and+McLean+Model+of+Information+Systems+Success%3A+A+Ten-Year+Update&rft.jtitle=Journal+of+Management+Information+Systems&rft.aulast=DeLone%2C+W.H.%3B+McLean%2C+E.R.&rft.au=DeLone%2C+W.H.%3B+McLean%2C+E.R.&rft.date=2003&rft.volume=19&rft.issue=4&rft.pages=9%E2%80%9330&rft_id=info:doi\/10.1080%2F07421222.2003.11045748&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JoshiPACS14-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JoshiPACS14_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Joshi, V.; Narra, V.R.; Joshi, K. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4090402\" target=\"_blank\">\"PACS administrators' and radiologists' perspective on the importance of features for PACS selection\"<\/a>. <i>Journal of Digital Imaging<\/i> <b>27<\/b> (4): 486-95. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-014-9682-3\" target=\"_blank\">10.1007\/s10278-014-9682-3<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4090402\/\" target=\"_blank\">PMC4090402<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24744278\" target=\"_blank\">24744278<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4090402\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4090402<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PACS+administrators%27+and+radiologists%27+perspective+on+the+importance+of+features+for+PACS+selection&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=Joshi%2C+V.%3B+Narra%2C+V.R.%3B+Joshi%2C+K.&rft.au=Joshi%2C+V.%3B+Narra%2C+V.R.%3B+Joshi%2C+K.&rft.date=2014&rft.volume=27&rft.issue=4&rft.pages=486-95&rft_id=info:doi\/10.1007%2Fs10278-014-9682-3&rft_id=info:pmc\/PMC4090402&rft_id=info:pmid\/24744278&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4090402&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WardTheAttitudes08-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WardTheAttitudes08_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ward, R.; Stevens, C.; Brentnall, P.; Briddon, J. (2008). \"The attitudes of health care staff to information technology: A comprehensive review of the research literature\". <i>Health Information and Libraries Journal<\/i> <b>25<\/b> (2): 81-97. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1471-1842.2008.00777.x\" target=\"_blank\">10.1111\/j.1471-1842.2008.00777.x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18494643\" target=\"_blank\">18494643<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+attitudes+of+health+care+staff+to+information+technology%3A+A+comprehensive+review+of+the+research+literature&rft.jtitle=Health+Information+and+Libraries+Journal&rft.aulast=Ward%2C+R.%3B+Stevens%2C+C.%3B+Brentnall%2C+P.%3B+Briddon%2C+J.&rft.au=Ward%2C+R.%3B+Stevens%2C+C.%3B+Brentnall%2C+P.%3B+Briddon%2C+J.&rft.date=2008&rft.volume=25&rft.issue=2&rft.pages=81-97&rft_id=info:doi\/10.1111%2Fj.1471-1842.2008.00777.x&rft_id=info:pmid\/18494643&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AdelekeICT15-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AdelekeICT15_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Adeleke, I.; Adejoke, A.; Achinbee, M. et al. (2015). \"ICT knowledge, Utilization and Perception among Healthcare Providers at National Hospital Abuja, Nigeria\". <i>American Journal of Health Research<\/i> <b>3<\/b> (1-1): 47-53. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.11648%2Fj.ajhr.s.2015030101.173\" target=\"_blank\">10.11648\/j.ajhr.s.2015030101.173<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ICT+knowledge%2C+Utilization+and+Perception+among+Healthcare+Providers+at+National+Hospital+Abuja%2C+Nigeria&rft.jtitle=American+Journal+of+Health+Research&rft.aulast=Adeleke%2C+I.%3B+Adejoke%2C+A.%3B+Achinbee%2C+M.+et+al.&rft.au=Adeleke%2C+I.%3B+Adejoke%2C+A.%3B+Achinbee%2C+M.+et+al.&rft.date=2015&rft.volume=3&rft.issue=1-1&rft.pages=47-53&rft_id=info:doi\/10.11648%2Fj.ajhr.s.2015030101.173&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SicotteVirtual10-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SicotteVirtual10_22-0\" rel=\"external_link\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-SicotteVirtual10_22-1\" rel=\"external_link\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sicotte, C.; Par\u00e9, G.; Bini, K.K. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3046687\" target=\"_blank\">\"Virtual organization of hospital medical imaging: A user satisfaction survey\"<\/a>. <i>Journal of Digital Imaging<\/i> <b>23<\/b> (6): 689-700. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-009-9220-x\" target=\"_blank\">10.1007\/s10278-009-9220-x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3046687\/\" target=\"_blank\">PMC3046687<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19588196\" target=\"_blank\">19588196<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3046687\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3046687<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Virtual+organization+of+hospital+medical+imaging%3A+A+user+satisfaction+survey&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=Sicotte%2C+C.%3B+Par%C3%A9%2C+G.%3B+Bini%2C+K.K.+et+al.&rft.au=Sicotte%2C+C.%3B+Par%C3%A9%2C+G.%3B+Bini%2C+K.K.+et+al.&rft.date=2010&rft.volume=23&rft.issue=6&rft.pages=689-700&rft_id=info:doi\/10.1007%2Fs10278-009-9220-x&rft_id=info:pmc\/PMC3046687&rft_id=info:pmid\/19588196&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3046687&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FolioQuantitative15-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FolioQuantitative15_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Folio, L.R.; Nelson, C.J.; Benjamin, M. et al. (2015). \"Quantitative Radiology Reporting in Oncology: Survey of Oncologists and Radiologists\". <i>American Journal of Roentgenology<\/i> <b>205<\/b> (3): W233-43. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2214%2FAJR.14.14054\" target=\"_blank\">10.2214\/AJR.14.14054<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26295661\" target=\"_blank\">26295661<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quantitative+Radiology+Reporting+in+Oncology%3A+Survey+of+Oncologists+and+Radiologists&rft.jtitle=American+Journal+of+Roentgenology&rft.aulast=Folio%2C+L.R.%3B+Nelson%2C+C.J.%3B+Benjamin%2C+M.+et+al.&rft.au=Folio%2C+L.R.%3B+Nelson%2C+C.J.%3B+Benjamin%2C+M.+et+al.&rft.date=2015&rft.volume=205&rft.issue=3&rft.pages=W233-43&rft_id=info:doi\/10.2214%2FAJR.14.14054&rft_id=info:pmid\/26295661&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SevensterImproved15-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SevensterImproved15_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sevenster, M.; Travis, A.R.; Ganesh, R.K. et al. (2015). \"Improved efficiency in clinical workflow of reporting measured oncology lesions via PACS-integrated lesion tracking tool\". <i>American Journal of Roentgenology<\/i> <b>204<\/b> (3): 576-83. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2214%2FAJR.14.12915\" target=\"_blank\">10.2214\/AJR.14.12915<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25714288\" target=\"_blank\">25714288<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improved+efficiency+in+clinical+workflow+of+reporting+measured+oncology+lesions+via+PACS-integrated+lesion+tracking+tool&rft.jtitle=American+Journal+of+Roentgenology&rft.aulast=Sevenster%2C+M.%3B+Travis%2C+A.R.%3B+Ganesh%2C+R.K.+et+al.&rft.au=Sevenster%2C+M.%3B+Travis%2C+A.R.%3B+Ganesh%2C+R.K.+et+al.&rft.date=2015&rft.volume=204&rft.issue=3&rft.pages=576-83&rft_id=info:doi\/10.2214%2FAJR.14.12915&rft_id=info:pmid\/25714288&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HsiaoCritical16-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HsiaoCritical16_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hsiao, J.L.; Chen, R.F. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4715302\" target=\"_blank\">\"Critical factors influencing physicians' intention to use computerized clinical practice guidelines: An integrative model of activity theory and the technology acceptance model\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>16<\/b>: 3. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12911-016-0241-3\" target=\"_blank\">10.1186\/s12911-016-0241-3<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4715302\/\" target=\"_blank\">PMC4715302<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26772169\" target=\"_blank\">26772169<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4715302\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4715302<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Critical+factors+influencing+physicians%27+intention+to+use+computerized+clinical+practice+guidelines%3A+An+integrative+model+of+activity+theory+and+the+technology+acceptance+model&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Hsiao%2C+J.L.%3B+Chen%2C+R.F.&rft.au=Hsiao%2C+J.L.%3B+Chen%2C+R.F.&rft.date=2016&rft.volume=16&rft.pages=3&rft_id=info:doi\/10.1186%2Fs12911-016-0241-3&rft_id=info:pmc\/PMC4715302&rft_id=info:pmid\/26772169&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4715302&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiuExploring15-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiuExploring15_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liu, C.F.; Cheng, T.J. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4333263\" target=\"_blank\">\"Exploring critical factors influencing physicians' acceptance of mobile electronic medical records based on the dual-factor model: A validation in Taiwan\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>15<\/b>: 4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12911-014-0125-3\" target=\"_blank\">10.1186\/s12911-014-0125-3<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4333263\/\" target=\"_blank\">PMC4333263<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25889506\" target=\"_blank\">25889506<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4333263\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4333263<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+critical+factors+influencing+physicians%27+acceptance+of+mobile+electronic+medical+records+based+on+the+dual-factor+model%3A+A+validation+in+Taiwan&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Liu%2C+C.F.%3B+Cheng%2C+T.J.&rft.au=Liu%2C+C.F.%3B+Cheng%2C+T.J.&rft.date=2015&rft.volume=15&rft.pages=4&rft_id=info:doi\/10.1186%2Fs12911-014-0125-3&rft_id=info:pmc\/PMC4333263&rft_id=info:pmid\/25889506&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4333263&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FloydStudy15-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FloydStudy15_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Floyd, D.M.; Trepp, E.R.; Ipaki, M.; Ng, C.K. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4441692\" target=\"_blank\">\"Study of Radiologic Technologists' Perceptions of Picture Archiving and Communication System (PACS) Competence and Educational Issues in Western Australia\"<\/a>. <i>Journal of Digital Imaging<\/i> <b>28<\/b> (3): 315\u201322. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-014-9765-1\" target=\"_blank\">10.1007\/s10278-014-9765-1<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4441692\/\" target=\"_blank\">PMC4441692<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25561072\" target=\"_blank\">25561072<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4441692\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4441692<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Study+of+Radiologic+Technologists%27+Perceptions+of+Picture+Archiving+and+Communication+System+%28PACS%29+Competence+and+Educational+Issues+in+Western+Australia&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=Floyd%2C+D.M.%3B+Trepp%2C+E.R.%3B+Ipaki%2C+M.%3B+Ng%2C+C.K.&rft.au=Floyd%2C+D.M.%3B+Trepp%2C+E.R.%3B+Ipaki%2C+M.%3B+Ng%2C+C.K.&rft.date=2015&rft.volume=28&rft.issue=3&rft.pages=315%E2%80%9322&rft_id=info:doi\/10.1007%2Fs10278-014-9765-1&rft_id=info:pmc\/PMC4441692&rft_id=info:pmid\/25561072&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4441692&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RayClin17-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RayClin17_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ray, K.N.; Felmet, K.A.; Hamilton, M.F. et al. (2017). \"Clinician Attitudes Toward Adoption of Pediatric Emergency Telemedicine in Rural Hospitals\". <i>Pediatric Emergency Care<\/i> <b>33<\/b> (4): 250-257. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FPEC.0000000000000583\" target=\"_blank\">10.1097\/PEC.0000000000000583<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26785087\" target=\"_blank\">26785087<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Clinician+Attitudes+Toward+Adoption+of+Pediatric+Emergency+Telemedicine+in+Rural+Hospitals&rft.jtitle=Pediatric+Emergency+Care&rft.aulast=Ray%2C+K.N.%3B+Felmet%2C+K.A.%3B+Hamilton%2C+M.F.+et+al.&rft.au=Ray%2C+K.N.%3B+Felmet%2C+K.A.%3B+Hamilton%2C+M.F.+et+al.&rft.date=2017&rft.volume=33&rft.issue=4&rft.pages=250-257&rft_id=info:doi\/10.1097%2FPEC.0000000000000583&rft_id=info:pmid\/26785087&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StraubTransfer01-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StraubTransfer01_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Straub, D.; Loch, K.; Hill, C. (2001). \"Transfer of Information Technology to the Arab World: A Test of Cultural Influence Modeling\". <i>Journal of Global Information Management<\/i> <b>9<\/b> (4): 6\u201328. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4018%2Fjgim.2001100101\" target=\"_blank\">10.4018\/jgim.2001100101<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Transfer+of+Information+Technology+to+the+Arab+World%3A+A+Test+of+Cultural+Influence+Modeling&rft.jtitle=Journal+of+Global+Information+Management&rft.aulast=Straub%2C+D.%3B+Loch%2C+K.%3B+Hill%2C+C.&rft.au=Straub%2C+D.%3B+Loch%2C+K.%3B+Hill%2C+C.&rft.date=2001&rft.volume=9&rft.issue=4&rft.pages=6%E2%80%9328&rft_id=info:doi\/10.4018%2Fjgim.2001100101&rfr_id=info:sid\/en.wikipedia.org:Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>CAD<\/b>: computer-assisted diagnosis\n<\/p><p><b>HIS<\/b>: health information system\n<\/p><p><b>ID<\/b>: identification\n<\/p><p><b>IS<\/b>: information system\n<\/p><p><b>PACS<\/b>: picture archiving and communication System\n<\/p><p><b>RIS<\/b>: radiology information system\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID was missing and was added to make the reference more useful. \n<\/p><p>Per the distribution agreement, the following copyright information is also being added: \n<\/p><p>\u00a9Ali Jassem Buabbas, Dawood Ameer Al-Shamali, Prem Sharma, Salwa Haidar, Hamza Al-Shawaf. Originally published in JMIR Medical Informatics (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/medinform.jmir.org\" target=\"_blank\">http:\/\/medinform.jmir.org<\/a>), 15.06.2016.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181454\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.731 seconds\nReal time usage: 0.764 seconds\nPreprocessor visited node count: 24743\/1000000\nPreprocessor generated node count: 35427\/1000000\nPost\u2010expand include size: 222437\/2097152 bytes\nTemplate argument size: 71633\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 704.507 1 - -total\n 85.51% 602.397 1 - Template:Reflist\n 76.10% 536.158 28 - Template:Cite_journal\n 75.20% 529.778 29 - Template:Citation\/core\n 11.34% 79.884 65 - Template:Citation\/identifier\n 8.55% 60.218 1 - Template:Infobox_journal_article\n 8.22% 57.903 1 - Template:Infobox\n 4.85% 34.178 80 - Template:Infobox\/row\n 4.20% 29.603 142 - Template:Hide_in_print\n 4.08% 28.730 29 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10130-0!*!0!!en!5!* and timestamp 20181214181453 and revision id 30662\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait\">https:\/\/www.limswiki.org\/index.php\/Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","016989f9a69b54fcd5805d9f8681d6fc_images":["https:\/\/www.limswiki.org\/images\/8\/80\/Fig1_Buabbas_JMIRMedInfo2016_4-2.png","https:\/\/www.limswiki.org\/images\/8\/89\/Fig2_Buabbas_JMIRMedInfo2016_4-2.png","https:\/\/www.limswiki.org\/images\/d\/dc\/Fig3_Buabbas_JMIRMedInfo2016_4-2.png","https:\/\/www.limswiki.org\/images\/3\/37\/Fig4_Buabbas_JMIRMedInfo2016_4-2.png","https:\/\/www.limswiki.org\/images\/a\/aa\/Fig5_Buabbas_JMIRMedInfo2016_4-2.png","https:\/\/www.limswiki.org\/images\/8\/8b\/Fig6_Buabbas_JMIRMedInfo2016_4-2.png"],"016989f9a69b54fcd5805d9f8681d6fc_timestamp":1544811293,"61508b4d2723dec78d6fb5e85ab3f5c1_type":"article","61508b4d2723dec78d6fb5e85ab3f5c1_title":"The state of open-source electronic health record projects: A software anthropology study (Alsaffar et al. 2017)","61508b4d2723dec78d6fb5e85ab3f5c1_url":"https:\/\/www.limswiki.org\/index.php\/Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study","61508b4d2723dec78d6fb5e85ab3f5c1_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:The state of open-source electronic health record projects: A software anthropology study\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nThe state of open-source electronic health record projects: A software anthropology studyJournal\n \nJMIR Medical InformaticsAuthor(s)\n \nAlsaffar, Mona; Yellowlees, Peter; Odor, Alberto; Hogarth, MichaelAuthor affiliation(s)\n \nUniversity of California-DavisPrimary contact\n \nEmail: mona_alsaffar [at] hotmail dot com; Phone: 1 9167348710Editors\n \nEysenbach, G.Year published\n \n2017Volume and issue\n \n5 (1)Page(s)\n \ne6DOI\n \n10.2196\/medinform.5783ISSN\n \n2291-9694Distribution license\n \nCreative Commons Attribution 2.0Website\n \nhttp:\/\/medinform.jmir.org\/2017\/1\/e6\/Download\n \nhttp:\/\/medinform.jmir.org\/2017\/1\/e6\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n\n2.1 Background \n2.2 Objectives \n\n\n3 Methods \n\n3.1 License type \n3.2 Downloads \n3.3 Development status \n3.4 Project age \n3.5 Programming language \n3.6 Spoken languages \n3.7 Supporting materials \n3.8 Top downloads according to countries \n3.9 Certified open source electronic health record \n3.10 Survey data collection \n\n\n4 Results \n\n4.1 Application data \n4.2 Survey data \n\n\n5 Discussion \n\n5.1 Principal findings \n5.2 Challenges \n\n5.2.1 Usability \n5.2.2 Interoperability \n5.2.3 Privacy and security \n5.2.4 Lack of financial and professional expertise \n5.2.5 Cost \n5.2.6 Skilled information technology personnel \n5.2.7 Limited functionality of clinical decision support \n5.2.8 Lack of liability and accountability \n\n\n5.3 Scope and limitations \n\n\n6 Conclusion \n7 Conflicts of interest \n8 Multimedia Appendix 1 \n9 References \n10 Abbreviations \n11 Notes \n\n\n\nAbstract \nBackground: Electronic health records (EHR) are a key tool in managing and storing patients\u2019 information. Currently, there are over 50 open-source EHR systems available. Functionality and usability are important factors for determining the success of any system. These factors are often a direct reflection of the domain knowledge and developers\u2019 motivations. However, few published studies have focused on the characteristics of free and open-source software (F\/OSS) EHR systems, and none to date have discussed the motivation, knowledge background, and demographic characteristics of the developers involved in open-source EHR projects.\nObjective: This study analyzed the characteristics of prevailing F\/OSS EHR systems and aimed to provide an understanding of the motivation, knowledge background, and characteristics of the developers.\nMethods: This study identified F\/OSS EHR projects on SourceForge and other websites from May to July 2014. Projects were classified and characterized by license type, downloads, programming languages, spoken languages, project age, development status, supporting materials, top downloads by country, and whether they were \"certified\" EHRs. Health care F\/OSS developers were also surveyed using an online survey.\nResults: At the time of the assessment, we uncovered 54 open source EHR projects, but only four of them had been successfully certified under the Office of the National Coordinator for Health Information Technology (ONC Health IT) Certification Program. In the majority of cases, the open-source EHR software was downloaded by users in the United States (64.07%, 148,666\/232,034), underscoring that there is a significant interest in EHR open-source applications in the United States. A survey of EHR open source developers was conducted, and a total of 103 developers responded to the online questionnaire. The majority of EHR F\/OSS developers (65.3%, 66\/101) are participating in F\/OSS projects as part of a paid activity, and only 25.7% (26\/101) of EHR F\/OSS developers are, or have been, health care providers in their careers. In addition, 45% (45\/99) of developers do not work in the health care field.\nConclusion: The research presented in this study highlights some challenges that may be hindering the future of health care F\/OSS. A minority of developers have been health care professionals, and only 55% (54\/99) work in the health care field. This undoubtedly hinders the functional design of F\/OSS EHR systems from exhibiting a competitive advantage over prevailing commercial EHR systems. Open-source software seems to be a significant interest to many; however, given that only four F\/OSS EHR systems are ONC-certified, this interest is unlikely to yield significant adoption of these systems in the United States. Although the Health Information Technology for Economic and Clinical Health (HITECH) Act was responsible for a substantial infusion of capital into the EHR marketplace, the lack of a corporate entity in most F\/OSS EHR projects translates to a marginal capacity to market the respective F\/OSS system and to navigate certification. This likely has further disadvantaged F\/OSS EHR adoption in the United States.\nKeywords: open source, electronic health record, SourceForge, developers, motivations \n\nIntroduction \nBackground \nThe medical field has been using open-source applications for almost 40 years.[1] Electronic health record (EHR) systems first appeared in the early 1960s.[2] The Computer Stored Ambulatory Record (COSTAR) system was the first F\/OSS EHR system and was originally developed to be used by the Harvard Community Health Plan. Although COSTAR was implemented in a number of institutions, it did not result in broad national adoption of EHRs at the time. Only the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 and its financial incentive program have resulted in broad adoption of EHRs in the United States.[3] F\/OSS EHR systems have been increasing in popularity over the period.[4]\nAlthough the HITECH incentive payments have increased adoption, EHR adoption continues to have obstacles.[5][6] One of the main obstacles continues to be affordability.[5] CDW Healthcare Physician Practice estimated the total cost of an EHR deployment at approximately USD $120,000 per physician in the first year after implementation, with annual recurring costs of USD $30,000 per physician.[7] Along with the financial cost, there is also the non-financial cost related to time spent to bring the system live and into full functional use.[7]\nOpen-source EHR may lessen financial barriers while also providing improved flexibility, given that they can be \"freely\" modified.[8] Many of the prevailing EHRs do not adhere to minimal usability testing standards[9], requiring continuous customization to meet the needs of the organization.[10] A KLAS study of 128 physicians on the current state of acute care EHRs found that no vendor scored high in usability.[11] Since open-source software can be freely modified and redistributed, this could reduce the cost of continuous customization to improve usability.[12] Open-source projects tend to also benefit from a higher degree of transparency about software anomalies (software bugs), leading to a higher degree of reliability over time. A common belief across the open source community, and often referred to as \"Linus Law,\" states \"given enough eyeballs, all bugs are shallow.\"[13] Unlike organizations who are dependent on a commercial vendor\u2019s prioritization of features and software release schedules, those implementing F\/OSS would have complete control over the timing of customization and deployment, allowing them to choose what functionality is available and when it will be available to their users.[14]\nF\/OSS does come with challenges as well. Although some commercial companies provide support for F\/OSS EHRs, the majority of the F\/OSS EHR projects do not have a support service one can purchase. This creates a major challenge in ensuring reliability, particularly when the original system has been customized by institutional programmers.[15] Those skeptical of F\/OSS EHR systems often highlight the potential dependency on volunteer developers[16] who do not guarantee technical support.[15] In addition, identifying a reliable source for version updates can be challenging.[15] Many organizations also fear that open-source projects can become inactive anytime, creating an acute need for substantial in-house software development expertise.[17] A majority of health care organizations do not typically have infrastructure to support software development, and they might not have information technology (IT) staff with expertise in managing the software development lifecycle (SDLC) for complex systems. Instead, the typical health care delivery organization\u2019s IT staff focuses on deploying and optimizing vendor software. \nDespite these disadvantages, F\/OSS software has been growing in terms of the number of projects. The Eighth Annual Future of Open Source Survey found that the number of F\/OSS projects doubled between 2012[18] and 2014.[19]\nThe core success of the open source movement depends on developers who contribute their knowledge and effort for free to the community. Developers are either unpaid volunteers, hobbyists[20], or employees who are paid to write code. A study of mainstream F\/OSS projects categorized developers\u2019 contribution into eight different roles: project leader, core member, active developers, peripheral developer, bug fixer, bug reporter, reader, and passive user.[4] As reflected in this categorization, there are a number of different roles for contributors and a significant amount of resources required to support a high-quality project. In large part, the developer community and their motivations are a key determinant of success or failure of an open-source project. Exploring these motivations is an important aspect of understanding a key success factor for F\/OSS EHR systems. The motivation-affecting factors for open source developers can be categorized as internal (cognitive) and external (social). Internal factors are comprised of motivation, altruism, and community identification. External motivation factors include future rewards (e.g., peer recognition), self-marketing, human capital, contribution as part of employment (i.e., being paid to contribute), and revenue from related products and services.[21] Generally, anything related to the joy of coding is considered intrinsic motivation, whereas extrinsic motivation is associated with receiving some benefit for the contribution. These factors have been explored in mainstream F\/OSS projects but have not yet been characterized in F\/OSS health care projects.\n\nObjectives \nThe objectives of this study were to canvass the current state of open-source EHR systems and to characterize the motivations, knowledge, and demographics of the developers.\n\nMethods \nTo find EHR F\/OSS projects, we used SourceForge, a widely used open source project repository, and Google, using the search terms \"electronic patient record,\" \"electronic health record,\" \"electronic medical records,\" and \"clinical information system.\" The search revealed hundreds of EHR F\/OSS projects, but only 54 of them were EHRs according to our study inclusion criteria. The following are two fundamental inclusion criteria used in the study[22]: \n\n The software had to be defined as an EHR, such that the project had to adhere to the functional definition of EHR. The HealthIT.gov website defines an EHR as a \"digital chart\" containing at least the medical and treatment history of the patients; and \n The software had to use an open-source license. Open-source software is defined as software without license restrictions on its redistribution, and the software can be freely modified.\nThe study was conducted for a three-month period starting May 2014. To understand the characteristics of the various EHR projects, we looked at license type, downloads, programming languages, spoken languages, project age, development status, supporting materials, top downloads by country, and whether they were \"certified\" EHRs.\n\nLicense type \nMany SourceForge applications are defined by their license type on the application homepage. In this study, the licenses were classified into permissive, restrictive, or highly restrictive.[22] The highly restrictive licenses, such as a general public license (GPL), allow free modification but request that any modification should be contributed back to the community under the same license. Highly restrictive licenses are used more in applications geared toward the end user (e.g., games).[23] License restrictions tend to affect who contributes and accessibility of the source code.[23]\n\nDownloads \nWe made an assumption that download frequency reflects the popularity of the software. This assumption is a commonly held belief in this research domain.[24] We looked at the download number in the last 12-month period on SourceForge. Around 47 projects (87%, 47\/54) in this study have information on downloads, a proxy for use of the software.\n\nDevelopment status \nDevelopment status shows the readiness of software for day to-day use. This study utilized the SourceForge classification for software readiness. In this study, six software stages were used: planning, pre-alpha, alpha, beta, production, stable, and mature. The software\u2019s status can influence a project\u2019s success and affects the interest of the users and developers.[24]\n\nProject age \nProject age represents the number of years since the project development started. The project\u2019s age, in addition to other factors, is positively related to its ability to attract more users and\/or resources, which affect the project\u2019s future sustainability.[25]\n\nProgramming language \nProgramming language for each software system was examined and classified according to whether one or multiple were used. One open source study suggests that using one common programming language affects the success of the software project.[26]\n\nSpoken languages \nThe projects were classified according to their spoken languages. One study proves that open source software popularity is related to the number of language versions available.[27]\n\nSupporting materials \nThe setup of the system is not always obvious, and in some cases it requires IT administration skills. The top 10 downloaded projects were analyzed, as these materials make the installation and usage of the system easier. The supporting materials included user guides, installation guides, and version demonstrations.\n\nTop downloads according to countries \nSourceForge provides important information about the software and gives the highest number of downloads for each EHR software system. The top 10 most downloaded projects were identified, in addition to the country that made the greatest number of downloads, to see if the adoption rate affects the number of downloads.\n\nCertified open source electronic health record \nUsing a certified EHR is a requirement for payments through the HITECH Act\u2019s EHR-incentive program. Certification of an EHR under the ONC program is an important success factor in the United States. The certified open-source EHR products and their specifications from the Centers for Medicare and Medicaid Services (CMS) website were examined (Table 1).\n\r\n\n\n\n\n\n\n\n\nTable 1. The Centers for Medicare and Medicaid Services-certified free and open source software electronic health record (EHR) applications\n\n\nProduct name\n\nOriginal practice type\n\nVendor\n\nProduct version number\n\nProduct classification\n\nCertification year\n\nCertification body\n\n\nOpenEMR\n\nAmbulatory\n\nOEMR\n\n4.1\n\nComplete EHR\n\n2011\n\nICSA Labs\n\n\nTolvenEMR\n\nAmbulatory, inpatient\n\nTolven, Inc.\n\n2.1\n\nComplete EHR, modular\n\n2011\n\nICSA Labs\n\n\nWorldVista\n\nAmbulatory, inpatient\n\nWorldVistA\n\n2.0\n\nN\/A\n\n2011\n\nInfoGrad\n\n\nClearHealth\n\nAmbulatory\n\nClearHealth, Inc.\n\n3.1.5\n\nComplete EHR\n\n2011\n\nInfoGrad\n\n\n\nSurvey data collection \nA survey was conducted in 2014 using the commercial survey tool SurveyGizmo (Boulder, CO). The survey consisted of 20 questions and took approximately five minutes to complete (Multimedia Appendix 1). The target audience included anyone who self-identified as a developer of health care F\/OSS. Our questionnaire was modeled after a similar survey developed to compare proprietary and open-source software in 2003.[28]\nAn announcement of the survey was published on 10 websites that focused on health care open source news and targeted health care developers. The announcement contained a brief summary of the main goal of the survey along with the author's names and their affiliations. About one week after this first announcement, the survey was distributed to 54 open-source project developers\u2019 mailing lists obtained from SourceForge. We reached out to project email addresses and asked them to distribute our survey to their mailing lists. In addition, we sent a personal survey invitation to specific developers who mentioned working in F\/OSS health care projects in their LinkedIn profile. The survey was posted for five weeks, and a total of 103 responses were collected.\n\nResults \nApplication data \nThe study revealed several key observations. At the time of the study, there were 54 open source EHR projects, but only four had been successfully certified under the Office of the National Coordinator for Health Information Technology (ONC Health IT) Certification Program. Nearly half of the projects (57%, 31\/54) used a restrictive license type, and approximately 57% (30\/54) used GPL. The data revealed that 52% (28\/54) of the projects were in production\/stable status, only 2 (4%, 2\/54) were in mature status, while 1 (2%, 1\/54) project was inactive (Table 2). There were 44 active projects at varying stages of development, while 10 had unspecified status.\n\n\n\n\n\n\n\nTable 2. Applications development status (N=54)\n\n\nDevelopment status\n\nFrequency, n (%)\n\n\nProduction\/stable\n\n28 (52%)\n\n\nUndetermined\n\n11 (20%)\n\n\nAlpha\n\n6 (11%)\n\n\nBeta\n\n6 (1%)\n\n\nMature\n\n2 (4%)\n\n\nPre-alpha\n\n1 (2%)\n\n\n\nAs one might expect, many open-source projects (46%, 25\/54) used one programming language. However, a large percentage (36%, 19\/54) used multiple programming languages. Approximately 18% (10\/54) of the projects did not indicate the use of a specific programming language. The analysis also showed that the number of downloads for the projects that were written using multiple programming languages (n=147,914) were higher than the projects using one programming language (n=115,299). Among those projects indicating a programming language, the \"PHP Hypertext Preprocessor,\" commonly known as PHP, was the leader, and it was used in 31% (17\/54) of the projects.\nThe OpenEMR project had the highest number of downloads (63,418 in a 12-month period) (Table 3). The data shows that the United States accounts for the majority of the downloads and constitutes 64.07% (148,666\/232,034) of the total downloads of open-source EHR projects on SourceForge. In total, 19% (10\/54) of open-source EHR systems have installation and user guides along with demonstration versions. The mean project age is seven years, with a range of two to 14 years.\n\r\n\n\n\n\n\n\n\n\nTable 3. The characteristics of the top 10 downloaded free and open source software electronic health record systems\n\n\nProduct name\n\nInstallation guide\n\nDemonstration\n\nUser guide\n\nTop downloads according to country\n\nDevelopment status\n\nStart year\n\nAge in years\n\n\nOpenEMR\n\nYes\n\nYes\n\nYes\n\nUnited States\n\nProduction\/stable\n\n2002\n\n12\n\n\nOpenMRS\n\nNo\n\nYes\n\nNo\n\nUnited States\n\nProduction\/stable\n\n2010\n\n4\n\n\nCare2X\n\nYes\n\nYes\n\nYes\n\nUnited States\n\nProduction\/stable\n\n2002\n\n12\n\n\nOpenClinic GA\n\nYes\n\nYes\n\nYes\n\nIndia\n\nProduction\/stable\n\n2010\n\n4\n\n\nOpen Hospital\n\nYes\n\nNo\n\nYes\n\nIndia\n\nProduction\/stable\n\n2006\n\n8\n\n\nFreeMED\n\nYes\n\nYes\n\nYes\n\nUnited States\n\nProduction\/stable\n\n2000\n\n14\n\n\nGNU Health\n\nYes\n\nYes\n\nYes\n\nIndia\n\nProduction\/stable\n\n2006\n\n8\n\n\nHOSxP\n\nYes\n\nYes\n\nNo\n\nThailand\n\nProduction\/stable\n\n2002\n\n12\n\n\nTolven Health Record\n\nYes\n\nYes\n\nYes\n\nFrance\n\nProduction\/stable\n\n2006\n\n8\n\n\nOSCAR McMaster\n\nYes\n\nYes\n\nYes\n\nCanada\n\nProduction\/stable\n\n2001\n\n13\n\n\n\nIn terms of project (spoken) language, a large number of projects were written in English (46%, 25\/54), but surprisingly 28% (15\/54) were being developed in one or more languages besides English. It was found that 11% (6\/54) of the projects did not specify a language (Table 4). Data shows that the number of downloads for projects with multiple spoken languages was very high (n=187,933); with a download rate three times the one-language projects (n=48,402). These numbers indicate a global interest in F\/OSS EHR development and prove that the number of language translations is positively related to project success and popularity.\n\r\n\n\n\n\n\n\n\n\nTable 4. Frequency of spoken languages (N=54)\n\n\nLanguage\n\nFrequency, n (%)\n\n\nEnglish\n\n25 (46%)\n\n\nNon-English\n\n4 (7%)\n\n\nMultiple (English plus other)\n\n15 (28%)\n\n\nMultiple (non-English)\n\n4 (7%)\n\n\nUnspecified\n\n6 (11%)\n\n\n\nSurvey data \nA total of 103 developer survey responses were successfully collected showing the developer\u2019s characteristics, background, and their reasons to contribute to health care open-source projects. Survey respondents were primarily male (94%, 94\/99) with an age range of 25 to 55 years old. The majority of developers were American (37%, 36\/98) and 42% (41\/98) lived in the United States. Employees made up 58% (58\/100) of the sample, and self-employed developers constituted 33.0% (33\/100). Approximately 58% (58\/99) of developers were married and 18.0% (18\/100) had children older than age six. As F\/OSS developers tend to be highly educated, many had graduate level (43.0%, 43\/100), professional level (18.0%, 18\/100), or undergraduate level (36.0%, 36\/100) education. Only 3.0% (3\/100) of F\/OSS developers reported not having formal education beyond high school.\nAs has been found in mainstream F\/OSS projects, the majority of health care F\/OSS developers participated in the projects as part of a paid activity. Nearly 34.7% (35\/101) of the contributors received direct payment for developing F\/OSS. Another 30.7% (31\/101) received direct payment for either managing or supporting F\/OSS while 34% (32\/94) received no payment at all for contributing to the projects. Most of the developers contributed by writing code during their off-work hours (41.6%, 42\/101) and (39.6%, 40\/101) during their work hours. Almost half of the respondents reported employer awareness of their F\/OSS work (42%, 42\/99). A number of respondents (32%, 32\/99) worked on F\/OSS as part of their employment. Some employers (17%, 17\/99) were unaware of the developers work on F\/OSS, but very few (3%, 3\/99) did not want them to contribute to F\/OSS development.\nContributors\u2019 efforts have been measured by the number of hours spent on a project per week. Respondents\u2019 answers show an average of 9.8 hours per week spent on F\/OSS projects. Health care open source developers were asked how many projects they had contributed to. Nearly 49.0% (50\/102) worked on one project, while 47% (48\/94) had worked on two to five open-source projects. The average number of projects was 1.6, with a maximum number of five projects.\nA significant number of respondents (47.0%, 47\/100) reported 75% to 100% of their code was included in a F\/OSS project. Around one third (43.0%, 43\/100) reported less than 25% of their code was included in the final project. The majority of contributors (50%, 48\/96) wrote between 500 and 5000 code lines, while only 30% (29\/96) wrote more than 10,000 lines. The average lines of code were 1776, with a maximum of 5000 lines.\nRespondents were asked to answer several questions regarding their F\/OSS project to analyze their opinions, motivations, and habits toward F\/OSS projects. Interestingly, developers had different motivations to participate in F\/OSS. The top reasons for contributing to a health care open-source project were based on enjoyment-related intrinsic motivation; the project was \"important and visible\" (47.5%, 48\/101) or \"technically interesting\" (47.5%, 48\/101) for them. Approximately 20.8% (21\/101) indicated community-based related intrinsic motivation as their reason to contribute and stated that they knew people working on the project.\nThe survey results confirmed that many developers start developing F\/OSS to give back to the community. Around 40% (39\/97) considered it important or very important to give back to the community and 38% (37\/96) considered the interaction with like-minded programmers to be important. A significant number of respondents were motivated to promote the mode of development and the ideal of freely modifiable software (47%, 47\/99), while the remaining developers were motivated to provide alternatives to proprietary software (53%, 50\/93). Some developers (27%, 25\/97) began developing F\/OSS software by modifying it to fit with their requirements or to fix the bugs in their existing software (32 %, 32\/97), and 26% (25\/96) of them were interested in learning how the program worked.\n\nDiscussion \nPrincipal findings \nOpen-source projects continue being started at a rapid pace, but sustainability of the projects appears to be a challenge. A study in 2005 found 45 F\/OSS EHR applications[29] compared to the 54 found in this study. This represents a 16.6% increase in the past nine years. However, most of the projects are different, suggesting sustainability is a significant challenge. Functional EHR systems are complex with many required subsystems and modules, requiring robust development and change management processes in order to achieve high-quality \"production-grade\" software for the mission-critical environment of a hospital or clinic. Certification is an added challenge in some markets.[30] Despite these challenges, F\/OSS EHR systems have been viewed as excellent options for community health clinics, small practices, and hospitals that are under-capitalized in the United States and overseas.[8]\nIn terms of open-source licenses, the majority of F\/OSS projects use GPL, which allows free modification but requires that any modification be contributed back to that open-source project under the same license. This creates a \"poison pill,\" making it difficult for commercial entities to integrate GPL open-source components into a module that also has proprietary software. This can be viewed as a \"restriction\" with regards to open-source licensing. Restrictiveness of the license also affects a number of contributors and accessibility of the source code.[23]\nWith regards to developers, this study found the average F\/OSS contributor is well-educated, young, and male; the F\/OSS community is male dominant, as women contributors are only 2%. The reasons for gender inequities remain unclear but may involve women facing hybrid discrimination from a F\/OSS community.[31] Our research shows marital status and having kids older than six years old increases the probability a person will volunteer.[32][33]\nOne of the most common aspects of F\/OSS is that they are based on voluntary efforts of developers. The goal of this research was to understand what motivates these developers to contribute to a health care F\/OSS project. Learning new skills and becoming a better programmer is one of the motivations this research found (35%, 34\/96), which is similar to previous findings about mainstream F\/OSS developers (36.5%).[28] Learning and acquiring new skills appears to be another important motivation for many developers.\nIntrinsic motivations, such as altruism, are high among health care developers; 47.5% (48\/101) of developers worked on the programs for this reason. Only 16% of developers in other research papers stated that altruism was their prime motivator.[21] Contrary to our research findings, some previous research papers[34][35] found that only a minority were motivated by extrinsic motivation, which involved payments for their participation in F\/OSS projects.\nOne of the unique findings in this research is that 74.2% (75\/101) of health care F\/OSS developers are not health care practitioners, and 45% (45\/99) do not work in the health care field. Being a developer outside of the health care field can be a core problem for the development of usable clinical software with a high degree of functionality. This may serve to explain why open-source EHRs have limited functionality today.\nAlthough the open-source projects solve licensing cost problems, there is a need for maintenance and implementation costs, which can be a barrier for organizations without health IT expertise. There are many open-source EHR options on SourceForge, but not all of them can fit in the clinical workflow in the United States or in certain hospitals.\n\nChallenges \nThe availability of motivated developers and the need to continuously improve EHR systems will likely mean F\/OSS EHR will continue to be part of the health care software landscape despite the many challenges these projects face today. Characterizing the challenges and benefits of adopting open-source EHR will be important in understanding the value of these systems.\n\nUsability \nThis study shows a potential weakness for F\/OSS EHR projects stemming from developer background and their ability to understand some of the nuances in health care workflows. A large fraction of developers (74.2%, 75\/101) were not health care practitioners, and few developers (54%, 54\/99) had worked in the health care field. A potential solution to this issue would be to have programs that give F\/OSS EHR developers direct access to providers and care venues.\n\nInteroperability \nMany hospitals would benefit greatly from integrated software, rather than a disparate group of systems. However, the F\/OSS option can be difficult to interface with commercial systems and may require personnel with multiple skill sets.[36]\n\nPrivacy and security \nMaintaining patient privacy through robust security is a critical aspect of any EHR system. A study of F\/OSS health care systems found the information is often not safeguarded with consent or privacy policies and offers limited protection against unauthorized access or release of information.[37] However, we did find one open-source EHR system (Tolven eCHR), which supported encryption of health care data at the row level in a relational database. This would make data unreadable even to the database administrator of the system. This is a high degree of security and not typical among EHR systems at the time.\n\nLack of financial and professional expertise \nLow acquisition, installation, and ownership costs make the F\/OSS system an excellent option for organizations with limited capital. However, in general, the health care environments that could benefit the most from F\/OSS EHR systems tend to have low information technology capital budgets and very limited access to health informatics professionals. These challenges may prevent the successful implementation of any health IT technology, including F\/OSS EHR.[36] Our study shows that developing countries adopted F\/OSS EHR systems, but the major adoptions were in North America after the HITECH Act. This suggests that even F\/OSS EHR system implementations require substantial financial and workforce resource capacity to succeed.\n\nCost \nAlthough F\/OSS software does not have licensing costs, effective implementation still requires skilled staff, time for installation, and time for learning the software. F\/OSS EHR implementation cost can be as high as the proprietary software because of the add-ons, consultation costs, and need for assistance.[36] An important aspect of the total cost of ownership of an EHR system is usability. Poor usability can directly impact the productivity of expensive health care providers and support personnel. The ability to freely modify F\/OSS EHR systems and optimize them for local use likely translates into lower overall cost.\n\nSkilled information technology personnel \nAdopting F\/OSS EHR requires a large number of IT employees with specific programming skills who understand the program well. A hospital that chooses to use F\/OSS EHR will need to hire developer IT staff and contract with extra IT vendors or consultant support.[36]\n\nLimited functionality of clinical decision support \nAt an operational level, F\/OSS software also presented reduced functions in decision support and knowledge management. Clinical reasoning, guidelines and protocols, quality assurance, and integrated care were rather limited or nonexistent in most applications.[37] Developers lack medical knowledge as the majority of them are not health care practitioners, which may affect the efficacy of clinical decision support functionality.\n\nLack of liability and accountability \nOne of the major shortcomings is the lack of liability and accountability in F\/OSS. Our study did not address this issue. Few studies of F\/OSS in health care have addressed the risks of using F\/OSS. F\/OSS EHR projects come without warranties regarding the development, release date, or fulfillment of functionality. F\/OSS EHR systems do not have a commercial entity providing support and tend to rely on a volunteer community. This can be a significant risk for a software system that supports the core business of a hospital.[36]\n\nScope and limitations \nData on F\/OSS EHR projects were primarily collected from SourceForge. Therefore, this study is a snapshot in time, with projects being added and deleted before or after data collection. This amount of \"churn\" reflects a dynamic software category where our findings may not apply in the future.\nNot all F\/OSS EHR projects were listed on SourceForge; some projects are managed through independent websites. Therefore, some of the projects on SourceForge were outdated, may have contained inaccurate information, and were a small representation of F\/OSS EHR projects as a whole. SourceForge was the main source for most of the data because it tends to be one of the preferred project management platforms for the open source movement and has numerous projects and registered developers.\nSurvey questions were mostly closed-ended which restricted respondents\u2019 answer options. However, we did accommodate all possible answers by providing an additional \"other\" option for some questions. There is a possibility that respondents misunderstood questions as there was no usability testing done on the survey prior to using it in the general F\/OSS EHR community; however, we did not receive any respondent requests to clarify any questions. Furthermore, the survey respondents do not reflect the entire population of open source developers, but we believe it is a reasonable representative of the target population.\n\nConclusion \nThis study highlights a number of important aspects of F\/OSS EHR. Open-source software systems seem to be important to some health care organizations; however, only four F\/OSS EHR systems are ONC-certified in the United States, which creates a barrier to broader use. Health care open-source software also currently lacks directed corporate or governmental support for sustainability and growth of these software programs. We hope this research underlines the challenges that hinder the future of F\/OSS and provides avenues for future research to study and improve adoption of F\/OSS systems in the United States.\n\nConflicts of interest \nNone declared.\n\nMultimedia Appendix 1 \nAdobe PDF File, 181KB\n\nReferences \n\n\n\u2191 McDonald, C.J.; Schadow, G.; Barnes, M. et al. (2003). \"Open source software in medical informatics -- Why, how and what\". International Journal of Medical Informatics 69 (2\u20133): 175\u201384. doi:10.1016\/S1386-5056(02)00104-1. PMID 12810121.   \n\n\u2191 Barnett, G.O.; Justice, N.S.; Somand, M.E. et al. (1978). \"COSTAR \u2014 A computer-based medical information system for ambulatory care\". Proceedings of the Annual Symposium on Computer Applications in Medical Care 1978: 486\u2013487. doi:10.1109\/SCAMC.1978.679955. PMC PMC2231744. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2231744 .   \n\n\u2191 Henry, J.W.; Pylypchuk, Y.; Searcy, T. et al. (May 2016). \"Adoption of Electronic Health Record Systems among U.S. Non-Federal Acute Care Hospitals: 2008-2015\". Health IT Dashboard. Office of the National Coordinator for Health Information Technology. https:\/\/dashboard.healthit.gov\/evaluations\/data-briefs\/non-federal-acute-care-hospital-ehr-adoption-2008-2015.php . Retrieved 21 September 2016 .   \n\n\u2191 4.0 4.1 Ye, Y.; Kishida, K. (2003). \"Toward an understanding of the motivation open source software developers\". Proceedings of the 25th International Conference on Software Engineering 2003: 419-29. ISBN 076951877X.   \n\n\u2191 5.0 5.1 Gans, D.; Kralewski, J.; Hammons, T. et al. (2005). \"Medical groups' adoption of electronic health records and information systems\". Health Affairs 24 (5): 1323-33. doi:10.1377\/hlthaff.24.5.1323. PMID 16162580.   \n\n\u2191 Ajami, S.; Bagheri-Tadi, T. (2013). \"Barriers for adopting electronic health records (EHRs) by physicians\". Acta Informatica Medica 21 (2): 129\u201334. doi:10.5455\/aim.2013.21.129-134. PMC PMC3766548. PMID 24058254. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3766548 .   \n\n\u2191 7.0 7.1 Fleming, N.S.; Culler, S.D.; McCorkle, R. et al. (2011). \"The financial and nonfinancial costs of implementing electronic health records in primary care practices\". Health Affairs 30 (3): 481-9. doi:10.1377\/hlthaff.2010.0768. PMID 21383367.   \n\n\u2191 8.0 8.1 Aminpour, F.; Sadoughi, F.; Ahamdi, M. (2014). \"Utilization of open source electronic health record around the world: A systematic review\". Journal of Research in Medical Sciences 19 (1): 57-64. PMC PMC3963324. PMID 24672566. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3963324 .   \n\n\u2191 Ratwani, R.M.; Benda, N.C.; Hettinger, A.Z. et al. (2015). \"Electronic health record vendor adherence to usability certification requirements and testing standards\". JAMA 314 (10): 1070-1. doi:10.1001\/jama.2015.8372. PMID 26348757.   \n\n\u2191 Briggs, B.; Carter-Templeton, H. (2014). \"Electronic health record customization: A quality improvement project\". Online Journal of Nursing Informatics 18 (3). http:\/\/www.himss.org\/electronic-health-record-customization-quality-improvement-project .   \n\n\u2191 Tate, C. (June 2015). \"AMDIS 2015\" (PDF). KLAS Enterprises, LLC. http:\/\/amdis.org\/wp-content\/uploads\/2015\/06\/Klas-Review_Tate1.pdf . Retrieved 21 September 2016 .   \n\n\u2191 Yellowlees, P.M.; Marks, S.L.; Hogarth, M. et al. (2008). \"Standards-based, open-source electronic health record systems: A desirable future for the U.S. health industry\". Telemedicine Journal and E-Health 14 (3): 284-8. doi:10.1089\/tmj.2007.0052. PMID 18570554.   \n\n\u2191 Raymond, E. (1999). \"The cathedral and the bazaar\". Knowledge, Technology & Policy 12 (3): 23\u201349. doi:10.1007\/s12130-999-1026-0.   \n\n\u2191 Kobayashi, S. (2012). \"Open Source Software Development on Medical Domain\". In Kalloniatis, C.. Modern Information Systems. InTech. pp. 1\u201316. doi:10.5772\/38117. ISBN 9789535106470.   \n\n\u2191 15.0 15.1 15.2 Maglogiannis, I. (2012). \"Towards the adoption of open source and open access electronic health record systems\". Journal of Healthcare Engineering 3 (1): 141-161. doi:10.1260\/2040-2295.3.1.141.   \n\n\u2191 Che\u0142kowski, T.; Gloor, P.; Jemielniak, D. (2016). \"Inequalities in open source software development: analysis of contributor's commits in Apache Software Foundation projects\". PLOS One 11 (4): e0152976. doi:10.1371\/journal.pone.0152976. PMC PMC4838325. PMID 27096157. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4838325 .   \n\n\u2191 Goulde, M.; Brown, E.; Rymer, J.; Hartzband, D. (March 2006). \"Open Source Software: A Primer for Health Care Leaders\". iHealth Reports. Forrester Consulting. pp. 31. http:\/\/www.chcf.org\/publications\/2006\/03\/open-source-software-a-primer-for-health-care-leaders . Retrieved 07 February 2017 .   \n\n\u2191 \"The Future of Open Source 2012 Survey Results\". North Bridge Venture Partners. 21 May 2012. http:\/\/www.northbridge.com\/2012-open-source-survey . Retrieved 21 September 2016 .   \n\n\u2191 \"The Future of Open Source 2014 Survey Results\". North Bridge Venture Partners. 2014. http:\/\/www.northbridge.com\/2014-future-open-source-survey-results-0 . Retrieved 21 September 2016 .   \n\n\u2191 Godfrey, M.W.; Tu, Q. (2000). \"Evolution in open source software: A case study\". Proceedings of the International Conference on Software Maintenance 2000: 131. doi:10.1109\/ICSM.2000.883030. ISBN 0769507530.   \n\n\u2191 21.0 21.1 Hars, A.; Ou, S. (2002). \"Working for free? Motivations for participating in open-source projects\". International Journal of Electronic Commerce 6 (3): 25\u201339. doi:10.1109\/HICSS.2001.927045.   \n\n\u2191 22.0 22.1 Sen, R.; Subramaniam, C.; Nelson, M. (2008). \"Determinants of the choice of open source software license\". Journal of Management Information Systems 25 (3): 207\u201340. doi:10.2753\/MIS0742-1222250306.   \n\n\u2191 23.0 23.1 23.2 Lerner, J.; Tirole, J. (2002). \"The scope of open source licensing\". Journal of Law, Economics and Organization 21 (1): 20\u201356. doi:10.3386\/w9363.   \n\n\u2191 24.0 24.1 Janamanchi, B.; Katsamakas, E.; Raghupathi, W. et al. (2009). \"The state and profile of open source software projects in health and medical informatics\". International Journal of Medical Informatics 78 (7): 457-72. doi:10.1016\/j.ijmedinf.2009.02.006. PMID 19321384.   \n\n\u2191 Chengalur-Smith, I.; Sidorova, A.; Daniel, S. (2010). \"Sustainability of free\/libre open source projects: A longitudinal study\". Journal of the Association for Information Systems 11 (11): 5. http:\/\/aisel.aisnet.org\/jais\/vol11\/iss11\/5\/ .   \n\n\u2191 Crowston, K.; Scozzi, B. (2002). \"Open source software projects as virtual organisations: Competency rallying for software development\". IEE Proceedings - Software 149 (1): 3\u201317. doi:10.1049\/ip-sen:20020197.   \n\n\u2191 Midha, V.; Palvia, P. (2012). \"Factors affecting the success of open source software\". Journal of Systems and Software 85 (4): 895-905. doi:10.1016\/j.jss.2011.11.010.   \n\n\u2191 28.0 28.1 Arora, S.; Waterman, A.; David, P.. \"Free\/Libre\/Open Source Software Group - Survey\". Stanford. http:\/\/web.stanford.edu\/group\/floss-us\/questionnaire\/FLOSS-US-Questionnaire.html . Retrieved 21 September 2016 .   \n\n\u2191 Hogarth, M.A.; Turner, S. (2005). \"A study of clinically related open source software projects\". AMIA Annual Symposium Proceedings 2005: 330\u20134. PMC PMC1560618. PMID 16779056. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1560618 .   \n\n\u2191 Karopka, T.; Schmuhl, H.; Demski, H. (2014). \"Free\/Libre open source software in health care: A review\". Healthcare Informatics Research 20 (1): 11\u201322. doi:10.4258\/hir.2014.20.1.11. PMC PMC3950260. PMID 24627814. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3950260 .   \n\n\u2191 Mahmod, M.; Yusof, S.; Dahalin, Z. (2010). \"Women contributions to open source software innovation: A social constructivist perspective\". 2010 International Symposium on Information Technology 2010: 1433-1438. doi:10.1109\/ITSIM.2010.5561496.   \n\n\u2191 Selbee, L.K.; Reed, P.B. (2000). \"Patterns of Volunteering Over the Life Cycle\" (PDF). Statistics Canada. http:\/\/www3.carleton.ca\/casr\/Patterns%20of%20Vol.pdf . Retrieved 21 September 2016 .   \n\n\u2191 Hayghe, H.V. (1991). \"Volunteers in the U.S.: Who donates the time?\". Monthly Labor Review 114 (2): 17\u201323. PMID 10110196.   \n\n\u2191 Lakhani, K.R.; Wolf, R. (2005). \"Chapter 1: Why Hackers Do What They Do: Understanding Motivation and Effort in Free\/Open Source Software Projects\". In Feller, J.; Fitzgerald, B.; Hissam, S.; Lakhani, K.R.. Perspectives on Free and Open Source Software. MIT Press. pp. 3\u201322. ISBN 9780262062466.   \n\n\u2191 Luthiger, B.; Jungwirth, C. (2007). \"Pervasive fun\". First Monday 12 (1). doi:10.5210\/fm.v12i1.1422.   \n\n\u2191 36.0 36.1 36.2 36.3 36.4 Schmuhl, H.; Heinze, O.; Bergh, B. (2013). \"Use of open source software in health care delivery - Results of a qualitative field study\". Yearbook of Medical Informatics 8: 107\u2013113. PMID 23974556.   \n\n\u2191 37.0 37.1 Flores Zuniga, A.E.; Win, K.T.; Susilo, W. (2010). \"Functionalities of free and open electronic health record systems\". International Journal of Technology Assessment in Health Care 26 (4): 382\u20139. doi:10.1017\/S0266462310001121. PMID 20974022.   \n\n\nAbbreviations \nCOSTAR: Computer Stored Ambulatory Record\nEHR: electronic health record\nF\/OSS: free and open source software\nGPL: general public license\nHITECH: Health Information Technology for Economic and Clinical Health\nIT: information technology\nONC: Office of the National Coordinator\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID and DOI was missing and were added to make the references more useful. \nPer the distribution agreement, the following copyright information is also being added: \n\u00a9Mona Alsaffar, Peter Yellowlees, Alberto Odor, Michael Hogarth. Originally published in JMIR Medical Informatics (http:\/\/medinform.jmir.org), 24.02.2017.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\">https:\/\/www.limswiki.org\/index.php\/Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 5 April 2017, at 16:14.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,165 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","61508b4d2723dec78d6fb5e85ab3f5c1_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_The_state_of_open-source_electronic_health_record_projects_A_software_anthropology_study skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:The state of open-source electronic health record projects: A software anthropology study<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">Electronic health records<\/a> (EHR) are a key tool in managing and storing patients\u2019 <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>. Currently, there are over 50 open-source EHR systems available. Functionality and usability are important factors for determining the success of any system. These factors are often a direct reflection of the domain knowledge and developers\u2019 motivations. However, few published studies have focused on the characteristics of <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"688dbf2b7fb58c69c6ad476828ee2c6a\">free and open-source software<\/a> (F\/OSS) EHR systems, and none to date have discussed the motivation, knowledge background, and demographic characteristics of the developers involved in open-source EHR projects.\n<\/p><p><b>Objective<\/b>: This study analyzed the characteristics of prevailing F\/OSS EHR systems and aimed to provide an understanding of the motivation, knowledge background, and characteristics of the developers.\n<\/p><p><b>Methods<\/b>: This study identified F\/OSS EHR projects on SourceForge and other websites from May to July 2014. Projects were classified and characterized by license type, downloads, programming languages, spoken languages, project age, development status, supporting materials, top downloads by country, and whether they were \"certified\" EHRs. Health care F\/OSS developers were also surveyed using an online survey.\n<\/p><p><b>Results<\/b>: At the time of the assessment, we uncovered 54 open source EHR projects, but only four of them had been successfully certified under the Office of the National Coordinator for Health Information Technology (ONC Health IT) Certification Program. In the majority of cases, the open-source EHR software was downloaded by users in the United States (64.07%, 148,666\/232,034), underscoring that there is a significant interest in EHR open-source applications in the United States. A survey of EHR open source developers was conducted, and a total of 103 developers responded to the online questionnaire. The majority of EHR F\/OSS developers (65.3%, 66\/101) are participating in F\/OSS projects as part of a paid activity, and only 25.7% (26\/101) of EHR F\/OSS developers are, or have been, health care providers in their careers. In addition, 45% (45\/99) of developers do not work in the health care field.\n<\/p><p><b>Conclusion<\/b>: The research presented in this study highlights some challenges that may be hindering the future of health care F\/OSS. A minority of developers have been health care professionals, and only 55% (54\/99) work in the health care field. This undoubtedly hinders the functional design of F\/OSS EHR systems from exhibiting a competitive advantage over prevailing commercial EHR systems. Open-source software seems to be a significant interest to many; however, given that only four F\/OSS EHR systems are ONC-certified, this interest is unlikely to yield significant adoption of these systems in the United States. Although the <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Information_Technology_for_Economic_and_Clinical_Health_Act\" title=\"Health Information Technology for Economic and Clinical Health Act\" target=\"_blank\" class=\"wiki-link\" data-key=\"89c9e20984bbda2e628b6d1d28ec3ad5\">Health Information Technology for Economic and Clinical Health<\/a> (HITECH) Act was responsible for a substantial infusion of capital into the EHR marketplace, the lack of a corporate entity in most F\/OSS EHR projects translates to a marginal capacity to market the respective F\/OSS system and to navigate certification. This likely has further disadvantaged F\/OSS EHR adoption in the United States.\n<\/p><p><b>Keywords<\/b>: open source, electronic health record, SourceForge, developers, motivations \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h3>\n<p>The medical field has been using open-source applications for almost 40 years.<sup id=\"rdp-ebb-cite_ref-McDonaldOpen03_1-0\" class=\"reference\"><a href=\"#cite_note-McDonaldOpen03-1\" rel=\"external_link\">[1]<\/a><\/sup> Electronic health record (EHR) systems first appeared in the early 1960s.<sup id=\"rdp-ebb-cite_ref-BarnettCOSTAR78_2-0\" class=\"reference\"><a href=\"#cite_note-BarnettCOSTAR78-2\" rel=\"external_link\">[2]<\/a><\/sup> The Computer Stored Ambulatory Record (COSTAR) system was the first F\/OSS EHR system and was originally developed to be used by the Harvard Community Health Plan. Although COSTAR was implemented in a number of institutions, it did not result in broad national adoption of EHRs at the time. Only the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 and its financial incentive program have resulted in broad adoption of EHRs in the United States.<sup id=\"rdp-ebb-cite_ref-HenryAdoption16_3-0\" class=\"reference\"><a href=\"#cite_note-HenryAdoption16-3\" rel=\"external_link\">[3]<\/a><\/sup> F\/OSS EHR systems have been increasing in popularity over the period.<sup id=\"rdp-ebb-cite_ref-YeToward03_4-0\" class=\"reference\"><a href=\"#cite_note-YeToward03-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/p><p>Although the HITECH incentive payments have increased adoption, EHR adoption continues to have obstacles.<sup id=\"rdp-ebb-cite_ref-GansMedical05_5-0\" class=\"reference\"><a href=\"#cite_note-GansMedical05-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AjamiBarriers13_6-0\" class=\"reference\"><a href=\"#cite_note-AjamiBarriers13-6\" rel=\"external_link\">[6]<\/a><\/sup> One of the main obstacles continues to be affordability.<sup id=\"rdp-ebb-cite_ref-GansMedical05_5-1\" class=\"reference\"><a href=\"#cite_note-GansMedical05-5\" rel=\"external_link\">[5]<\/a><\/sup> CDW Healthcare Physician Practice estimated the total cost of an EHR deployment at approximately USD $120,000 per physician in the first year after implementation, with annual recurring costs of USD $30,000 per physician.<sup id=\"rdp-ebb-cite_ref-FlemingTheFin11_7-0\" class=\"reference\"><a href=\"#cite_note-FlemingTheFin11-7\" rel=\"external_link\">[7]<\/a><\/sup> Along with the financial cost, there is also the non-financial cost related to time spent to bring the system live and into full functional use.<sup id=\"rdp-ebb-cite_ref-FlemingTheFin11_7-1\" class=\"reference\"><a href=\"#cite_note-FlemingTheFin11-7\" rel=\"external_link\">[7]<\/a><\/sup>\n<\/p><p>Open-source EHR may lessen financial barriers while also providing improved flexibility, given that they can be \"freely\" modified.<sup id=\"rdp-ebb-cite_ref-AminpourUtil14_8-0\" class=\"reference\"><a href=\"#cite_note-AminpourUtil14-8\" rel=\"external_link\">[8]<\/a><\/sup> Many of the prevailing EHRs do not adhere to minimal usability testing standards<sup id=\"rdp-ebb-cite_ref-RatwaniElect15_9-0\" class=\"reference\"><a href=\"#cite_note-RatwaniElect15-9\" rel=\"external_link\">[9]<\/a><\/sup>, requiring continuous customization to meet the needs of the organization.<sup id=\"rdp-ebb-cite_ref-BriggsElect14_10-0\" class=\"reference\"><a href=\"#cite_note-BriggsElect14-10\" rel=\"external_link\">[10]<\/a><\/sup> A KLAS study of 128 physicians on the current state of acute care EHRs found that no vendor scored high in usability.<sup id=\"rdp-ebb-cite_ref-11\" class=\"reference\"><a href=\"#cite_note-11\" rel=\"external_link\">[11]<\/a><\/sup> Since open-source software can be freely modified and redistributed, this could reduce the cost of continuous customization to improve usability.<sup id=\"rdp-ebb-cite_ref-YellowleesStandards08_12-0\" class=\"reference\"><a href=\"#cite_note-YellowleesStandards08-12\" rel=\"external_link\">[12]<\/a><\/sup> Open-source projects tend to also benefit from a higher degree of transparency about software anomalies (software bugs), leading to a higher degree of reliability over time. A common belief across the open source community, and often referred to as \"Linus Law,\" states \"given enough eyeballs, all bugs are shallow.\"<sup id=\"rdp-ebb-cite_ref-RaymondTheCath199_13-0\" class=\"reference\"><a href=\"#cite_note-RaymondTheCath199-13\" rel=\"external_link\">[13]<\/a><\/sup> Unlike organizations who are dependent on a commercial vendor\u2019s prioritization of features and software release schedules, those implementing F\/OSS would have complete control over the timing of customization and deployment, allowing them to choose what functionality is available and when it will be available to their users.<sup id=\"rdp-ebb-cite_ref-KobayashiOpen12_14-0\" class=\"reference\"><a href=\"#cite_note-KobayashiOpen12-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/p><p>F\/OSS does come with challenges as well. Although some commercial companies provide support for F\/OSS EHRs, the majority of the F\/OSS EHR projects do not have a support service one can purchase. This creates a major challenge in ensuring reliability, particularly when the original system has been customized by institutional programmers.<sup id=\"rdp-ebb-cite_ref-MaglogiannisTowards12_15-0\" class=\"reference\"><a href=\"#cite_note-MaglogiannisTowards12-15\" rel=\"external_link\">[15]<\/a><\/sup> Those skeptical of F\/OSS EHR systems often highlight the potential dependency on volunteer developers<sup id=\"rdp-ebb-cite_ref-ChelkowskiIneq16_16-0\" class=\"reference\"><a href=\"#cite_note-ChelkowskiIneq16-16\" rel=\"external_link\">[16]<\/a><\/sup> who do not guarantee technical support.<sup id=\"rdp-ebb-cite_ref-MaglogiannisTowards12_15-1\" class=\"reference\"><a href=\"#cite_note-MaglogiannisTowards12-15\" rel=\"external_link\">[15]<\/a><\/sup> In addition, identifying a reliable source for version updates can be challenging.<sup id=\"rdp-ebb-cite_ref-MaglogiannisTowards12_15-2\" class=\"reference\"><a href=\"#cite_note-MaglogiannisTowards12-15\" rel=\"external_link\">[15]<\/a><\/sup> Many organizations also fear that open-source projects can become inactive anytime, creating an acute need for substantial in-house software development expertise.<sup id=\"rdp-ebb-cite_ref-GouldeOpen06_17-0\" class=\"reference\"><a href=\"#cite_note-GouldeOpen06-17\" rel=\"external_link\">[17]<\/a><\/sup> A majority of health care organizations do not typically have infrastructure to support software development, and they might not have information technology (IT) staff with expertise in managing the software development lifecycle (SDLC) for complex systems. Instead, the typical health care delivery organization\u2019s IT staff focuses on deploying and optimizing vendor software. \n<\/p><p>Despite these disadvantages, F\/OSS software has been growing in terms of the number of projects. The Eighth Annual Future of Open Source Survey found that the number of F\/OSS projects doubled between 2012<sup id=\"rdp-ebb-cite_ref-NBVPTheFuture12_18-0\" class=\"reference\"><a href=\"#cite_note-NBVPTheFuture12-18\" rel=\"external_link\">[18]<\/a><\/sup> and 2014.<sup id=\"rdp-ebb-cite_ref-NBVPTheFuture14_19-0\" class=\"reference\"><a href=\"#cite_note-NBVPTheFuture14-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/p><p>The core success of the open source movement depends on developers who contribute their knowledge and effort for free to the community. Developers are either unpaid volunteers, hobbyists<sup id=\"rdp-ebb-cite_ref-GodfreyEvolution00_20-0\" class=\"reference\"><a href=\"#cite_note-GodfreyEvolution00-20\" rel=\"external_link\">[20]<\/a><\/sup>, or employees who are paid to write code. A study of mainstream F\/OSS projects categorized developers\u2019 contribution into eight different roles: project leader, core member, active developers, peripheral developer, bug fixer, bug reporter, reader, and passive user.<sup id=\"rdp-ebb-cite_ref-YeToward03_4-1\" class=\"reference\"><a href=\"#cite_note-YeToward03-4\" rel=\"external_link\">[4]<\/a><\/sup> As reflected in this categorization, there are a number of different roles for contributors and a significant amount of resources required to support a high-quality project. In large part, the developer community and their motivations are a key determinant of success or failure of an open-source project. Exploring these motivations is an important aspect of understanding a key success factor for F\/OSS EHR systems. The motivation-affecting factors for open source developers can be categorized as internal (cognitive) and external (social). Internal factors are comprised of motivation, altruism, and community identification. External motivation factors include future rewards (e.g., peer recognition), self-marketing, human capital, contribution as part of employment (i.e., being paid to contribute), and revenue from related products and services.<sup id=\"rdp-ebb-cite_ref-HarsWorking02_21-0\" class=\"reference\"><a href=\"#cite_note-HarsWorking02-21\" rel=\"external_link\">[21]<\/a><\/sup> Generally, anything related to the joy of coding is considered intrinsic motivation, whereas extrinsic motivation is associated with receiving some benefit for the contribution. These factors have been explored in mainstream F\/OSS projects but have not yet been characterized in F\/OSS health care projects.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Objectives\">Objectives<\/span><\/h3>\n<p>The objectives of this study were to canvass the current state of open-source EHR systems and to characterize the motivations, knowledge, and demographics of the developers.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>To find EHR F\/OSS projects, we used SourceForge, a widely used open source project repository, and Google, using the search terms \"electronic patient record,\" \"electronic health record,\" \"<a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" target=\"_blank\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical records<\/a>,\" and \"clinical information system.\" The search revealed hundreds of EHR F\/OSS projects, but only 54 of them were EHRs according to our study inclusion criteria. The following are two fundamental inclusion criteria used in the study<sup id=\"rdp-ebb-cite_ref-SenDeterm08_22-0\" class=\"reference\"><a href=\"#cite_note-SenDeterm08-22\" rel=\"external_link\">[22]<\/a><\/sup>: \n<\/p>\n<ol><li> The software had to be defined as an EHR, such that the project had to adhere to the functional definition of EHR. The HealthIT.gov website defines an EHR as a \"digital chart\" containing at least the medical and treatment history of the patients; and <\/li>\n<li> The software had to use an open-source license. Open-source software is defined as software without license restrictions on its redistribution, and the software can be freely modified.<\/li><\/ol>\n<p>The study was conducted for a three-month period starting May 2014. To understand the characteristics of the various EHR projects, we looked at license type, downloads, programming languages, spoken languages, project age, development status, supporting materials, top downloads by country, and whether they were \"certified\" EHRs.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"License_type\">License type<\/span><\/h3>\n<p>Many SourceForge applications are defined by their license type on the application homepage. In this study, the licenses were classified into permissive, restrictive, or highly restrictive.<sup id=\"rdp-ebb-cite_ref-SenDeterm08_22-1\" class=\"reference\"><a href=\"#cite_note-SenDeterm08-22\" rel=\"external_link\">[22]<\/a><\/sup> The highly restrictive licenses, such as a general public license (GPL), allow free modification but request that any modification should be contributed back to the community under the same license. Highly restrictive licenses are used more in applications geared toward the end user (e.g., games).<sup id=\"rdp-ebb-cite_ref-LernerTheScope02_23-0\" class=\"reference\"><a href=\"#cite_note-LernerTheScope02-23\" rel=\"external_link\">[23]<\/a><\/sup> License restrictions tend to affect who contributes and accessibility of the source code.<sup id=\"rdp-ebb-cite_ref-LernerTheScope02_23-1\" class=\"reference\"><a href=\"#cite_note-LernerTheScope02-23\" rel=\"external_link\">[23]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Downloads\">Downloads<\/span><\/h3>\n<p>We made an assumption that download frequency reflects the popularity of the software. This assumption is a commonly held belief in this research domain.<sup id=\"rdp-ebb-cite_ref-JanamanchiTheState09_24-0\" class=\"reference\"><a href=\"#cite_note-JanamanchiTheState09-24\" rel=\"external_link\">[24]<\/a><\/sup> We looked at the download number in the last 12-month period on SourceForge. Around 47 projects (87%, 47\/54) in this study have information on downloads, a proxy for use of the software.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Development_status\">Development status<\/span><\/h3>\n<p>Development status shows the readiness of software for day to-day use. This study utilized the SourceForge classification for software readiness. In this study, six software stages were used: planning, pre-alpha, alpha, beta, production, stable, and mature. The software\u2019s status can influence a project\u2019s success and affects the interest of the users and developers.<sup id=\"rdp-ebb-cite_ref-JanamanchiTheState09_24-1\" class=\"reference\"><a href=\"#cite_note-JanamanchiTheState09-24\" rel=\"external_link\">[24]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Project_age\">Project age<\/span><\/h3>\n<p>Project age represents the number of years since the project development started. The project\u2019s age, in addition to other factors, is positively related to its ability to attract more users and\/or resources, which affect the project\u2019s future sustainability.<sup id=\"rdp-ebb-cite_ref-Chengalur-SmithSust10_25-0\" class=\"reference\"><a href=\"#cite_note-Chengalur-SmithSust10-25\" rel=\"external_link\">[25]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Programming_language\">Programming language<\/span><\/h3>\n<p>Programming language for each software system was examined and classified according to whether one or multiple were used. One open source study suggests that using one common programming language affects the success of the software project.<sup id=\"rdp-ebb-cite_ref-CrowstonOpen02_26-0\" class=\"reference\"><a href=\"#cite_note-CrowstonOpen02-26\" rel=\"external_link\">[26]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Spoken_languages\">Spoken languages<\/span><\/h3>\n<p>The projects were classified according to their spoken languages. One study proves that open source software popularity is related to the number of language versions available.<sup id=\"rdp-ebb-cite_ref-MidhaFactors12_27-0\" class=\"reference\"><a href=\"#cite_note-MidhaFactors12-27\" rel=\"external_link\">[27]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Supporting_materials\">Supporting materials<\/span><\/h3>\n<p>The setup of the system is not always obvious, and in some cases it requires IT administration skills. The top 10 downloaded projects were analyzed, as these materials make the installation and usage of the system easier. The supporting materials included user guides, installation guides, and version demonstrations.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Top_downloads_according_to_countries\">Top downloads according to countries<\/span><\/h3>\n<p>SourceForge provides important information about the software and gives the highest number of downloads for each EHR software system. The top 10 most downloaded projects were identified, in addition to the country that made the greatest number of downloads, to see if the adoption rate affects the number of downloads.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Certified_open_source_electronic_health_record\">Certified open source electronic health record<\/span><\/h3>\n<p>Using a certified EHR is a requirement for payments through the HITECH Act\u2019s EHR-incentive program. Certification of an EHR under the ONC program is an important success factor in the United States. The certified open-source EHR products and their specifications from the <a href=\"https:\/\/www.limswiki.org\/index.php\/Centers_for_Medicare_and_Medicaid_Services\" title=\"Centers for Medicare and Medicaid Services\" target=\"_blank\" class=\"wiki-link\" data-key=\"654b4449e4816e190325b420c264df1a\">Centers for Medicare and Medicaid Services<\/a> (CMS) website were examined (Table 1).\n<\/p><p><br \/>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><b>Table 1.<\/b> The Centers for Medicare and Medicaid Services-certified free and open source software electronic health record (EHR) applications\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Product name\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Original practice type\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Vendor\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Product version number\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Product classification\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Certification year\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Certification body\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/OpenEMR\" title=\"OpenEMR\" target=\"_blank\" class=\"wiki-link\" data-key=\"ad61bf020caf7f6dd1c1ea1e59c9fe9a\">OpenEMR<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ambulatory\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">OEMR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Complete EHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2011\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ICSA Labs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TolvenEMR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ambulatory, inpatient\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tolven, Inc.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Complete EHR, modular\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2011\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ICSA Labs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WorldVista\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ambulatory, inpatient\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WorldVistA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\/A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2011\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">InfoGrad\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/ClearHealth\" title=\"ClearHealth\" target=\"_blank\" class=\"wiki-link\" data-key=\"e170c2235c416ee8fa82d4976e7094a6\">ClearHealth<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ambulatory\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ClearHealth, Inc.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.1.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Complete EHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2011\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">InfoGrad\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Survey_data_collection\">Survey data collection<\/span><\/h3>\n<p>A survey was conducted in 2014 using the commercial survey tool SurveyGizmo (Boulder, CO). The survey consisted of 20 questions and took approximately five minutes to complete (Multimedia Appendix 1). The target audience included anyone who self-identified as a developer of health care F\/OSS. Our questionnaire was modeled after a similar survey developed to compare proprietary and open-source software in 2003.<sup id=\"rdp-ebb-cite_ref-AroraFree_28-0\" class=\"reference\"><a href=\"#cite_note-AroraFree-28\" rel=\"external_link\">[28]<\/a><\/sup>\n<\/p><p>An announcement of the survey was published on 10 websites that focused on health care open source news and targeted health care developers. The announcement contained a brief summary of the main goal of the survey along with the author's names and their affiliations. About one week after this first announcement, the survey was distributed to 54 open-source project developers\u2019 mailing lists obtained from SourceForge. We reached out to project email addresses and asked them to distribute our survey to their mailing lists. In addition, we sent a personal survey invitation to specific developers who mentioned working in F\/OSS health care projects in their LinkedIn profile. The survey was posted for five weeks, and a total of 103 responses were collected.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Application_data\">Application data<\/span><\/h3>\n<p>The study revealed several key observations. At the time of the study, there were 54 open source EHR projects, but only four had been successfully certified under the Office of the National Coordinator for Health Information Technology (ONC Health IT) Certification Program. Nearly half of the projects (57%, 31\/54) used a restrictive license type, and approximately 57% (30\/54) used GPL. The data revealed that 52% (28\/54) of the projects were in production\/stable status, only 2 (4%, 2\/54) were in mature status, while 1 (2%, 1\/54) project was inactive (Table 2). There were 44 active projects at varying stages of development, while 10 had unspecified status.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 2.<\/b> Applications development status (N=54)\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Development status\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Frequency, n (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28 (52%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Undetermined\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11 (20%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Alpha\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6 (11%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Beta\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6 (1%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mature\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (4%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pre-alpha\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (2%)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As one might expect, many open-source projects (46%, 25\/54) used one programming language. However, a large percentage (36%, 19\/54) used multiple programming languages. Approximately 18% (10\/54) of the projects did not indicate the use of a specific programming language. The analysis also showed that the number of downloads for the projects that were written using multiple programming languages (n=147,914) were higher than the projects using one programming language (n=115,299). Among those projects indicating a programming language, the \"PHP Hypertext Preprocessor,\" commonly known as PHP, was the leader, and it was used in 31% (17\/54) of the projects.\n<\/p><p>The <a href=\"https:\/\/www.limswiki.org\/index.php\/OpenEMR\" title=\"OpenEMR\" target=\"_blank\" class=\"wiki-link\" data-key=\"ad61bf020caf7f6dd1c1ea1e59c9fe9a\">OpenEMR<\/a> project had the highest number of downloads (63,418 in a 12-month period) (Table 3). The data shows that the United States accounts for the majority of the downloads and constitutes 64.07% (148,666\/232,034) of the total downloads of open-source EHR projects on SourceForge. In total, 19% (10\/54) of open-source EHR systems have installation and user guides along with demonstration versions. The mean project age is seven years, with a range of two to 14 years.\n<\/p><p><br \/>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><b>Table 3.<\/b> The characteristics of the top 10 downloaded free and open source software electronic health record systems\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Product name\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Installation guide\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Demonstration\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">User guide\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Top downloads according to country\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Development status\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Start year\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Age in years\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/OpenEMR\" title=\"OpenEMR\" target=\"_blank\" class=\"wiki-link\" data-key=\"ad61bf020caf7f6dd1c1ea1e59c9fe9a\">OpenEMR<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">United States\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2002\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/OpenMRS\" title=\"OpenMRS\" target=\"_blank\" class=\"wiki-link\" data-key=\"bdb797cd304c9ac66336f395c2bb6dbd\">OpenMRS<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">United States\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2010\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Care2X\" title=\"Care2X\" target=\"_blank\" class=\"wiki-link\" data-key=\"ddadc10e3b4b9108af55969fd50561b4\">Care2X<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">United States\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2002\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/OpenClinic_GA\" title=\"OpenClinic GA\" target=\"_blank\" class=\"wiki-link\" data-key=\"3dba47138f97f4a956e9216f3f34b1b1\">OpenClinic GA<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">India\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2010\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Open_Hospital\" title=\"Open Hospital\" target=\"_blank\" class=\"wiki-link\" data-key=\"5c1b0c23669a9e6dc1920875ae069bab\">Open Hospital<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">India\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2006\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">FreeMED\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">United States\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/GNU_Health\" title=\"GNU Health\" target=\"_blank\" class=\"wiki-link\" data-key=\"6f7172a563b254c7b34c3805b4a845e7\">GNU Health<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">India\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2006\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HOSxP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Thailand\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2002\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tolven Health Record\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">France\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2006\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/OSCAR\" title=\"OSCAR\" target=\"_blank\" class=\"wiki-link\" data-key=\"2e731490fe7e8eaf5782e3e35742c251\">OSCAR McMaster<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Canada\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Production\/stable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In terms of project (spoken) language, a large number of projects were written in English (46%, 25\/54), but surprisingly 28% (15\/54) were being developed in one or more languages besides English. It was found that 11% (6\/54) of the projects did not specify a language (Table 4). Data shows that the number of downloads for projects with multiple spoken languages was very high (n=187,933); with a download rate three times the one-language projects (n=48,402). These numbers indicate a global interest in F\/OSS EHR development and prove that the number of language translations is positively related to project success and popularity.\n<\/p><p><br \/>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 4.<\/b> Frequency of spoken languages (N=54)\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Language\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Frequency, n (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">English\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25 (46%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Non-English\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (7%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple (English plus other)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15 (28%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple (non-English)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (7%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Unspecified\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6 (11%)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Survey_data\">Survey data<\/span><\/h3>\n<p>A total of 103 developer survey responses were successfully collected showing the developer\u2019s characteristics, background, and their reasons to contribute to health care open-source projects. Survey respondents were primarily male (94%, 94\/99) with an age range of 25 to 55 years old. The majority of developers were American (37%, 36\/98) and 42% (41\/98) lived in the United States. Employees made up 58% (58\/100) of the sample, and self-employed developers constituted 33.0% (33\/100). Approximately 58% (58\/99) of developers were married and 18.0% (18\/100) had children older than age six. As F\/OSS developers tend to be highly educated, many had graduate level (43.0%, 43\/100), professional level (18.0%, 18\/100), or undergraduate level (36.0%, 36\/100) education. Only 3.0% (3\/100) of F\/OSS developers reported not having formal education beyond high school.\n<\/p><p>As has been found in mainstream F\/OSS projects, the majority of health care F\/OSS developers participated in the projects as part of a paid activity. Nearly 34.7% (35\/101) of the contributors received direct payment for developing F\/OSS. Another 30.7% (31\/101) received direct payment for either managing or supporting F\/OSS while 34% (32\/94) received no payment at all for contributing to the projects. Most of the developers contributed by writing code during their off-work hours (41.6%, 42\/101) and (39.6%, 40\/101) during their work hours. Almost half of the respondents reported employer awareness of their F\/OSS work (42%, 42\/99). A number of respondents (32%, 32\/99) worked on F\/OSS as part of their employment. Some employers (17%, 17\/99) were unaware of the developers work on F\/OSS, but very few (3%, 3\/99) did not want them to contribute to F\/OSS development.\n<\/p><p>Contributors\u2019 efforts have been measured by the number of hours spent on a project per week. Respondents\u2019 answers show an average of 9.8 hours per week spent on F\/OSS projects. Health care open source developers were asked how many projects they had contributed to. Nearly 49.0% (50\/102) worked on one project, while 47% (48\/94) had worked on two to five open-source projects. The average number of projects was 1.6, with a maximum number of five projects.\n<\/p><p>A significant number of respondents (47.0%, 47\/100) reported 75% to 100% of their code was included in a F\/OSS project. Around one third (43.0%, 43\/100) reported less than 25% of their code was included in the final project. The majority of contributors (50%, 48\/96) wrote between 500 and 5000 code lines, while only 30% (29\/96) wrote more than 10,000 lines. The average lines of code were 1776, with a maximum of 5000 lines.\n<\/p><p>Respondents were asked to answer several questions regarding their F\/OSS project to analyze their opinions, motivations, and habits toward F\/OSS projects. Interestingly, developers had different motivations to participate in F\/OSS. The top reasons for contributing to a health care open-source project were based on enjoyment-related intrinsic motivation; the project was \"important and visible\" (47.5%, 48\/101) or \"technically interesting\" (47.5%, 48\/101) for them. Approximately 20.8% (21\/101) indicated community-based related intrinsic motivation as their reason to contribute and stated that they knew people working on the project.\n<\/p><p>The survey results confirmed that many developers start developing F\/OSS to give back to the community. Around 40% (39\/97) considered it important or very important to give back to the community and 38% (37\/96) considered the interaction with like-minded programmers to be important. A significant number of respondents were motivated to promote the mode of development and the ideal of freely modifiable software (47%, 47\/99), while the remaining developers were motivated to provide alternatives to proprietary software (53%, 50\/93). Some developers (27%, 25\/97) began developing F\/OSS software by modifying it to fit with their requirements or to fix the bugs in their existing software (32 %, 32\/97), and 26% (25\/96) of them were interested in learning how the program worked.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Principal_findings\">Principal findings<\/span><\/h3>\n<p>Open-source projects continue being started at a rapid pace, but sustainability of the projects appears to be a challenge. A study in 2005 found 45 F\/OSS EHR applications<sup id=\"rdp-ebb-cite_ref-HogarthAStudy05_29-0\" class=\"reference\"><a href=\"#cite_note-HogarthAStudy05-29\" rel=\"external_link\">[29]<\/a><\/sup> compared to the 54 found in this study. This represents a 16.6% increase in the past nine years. However, most of the projects are different, suggesting sustainability is a significant challenge. Functional EHR systems are complex with many required subsystems and modules, requiring robust development and change management processes in order to achieve high-quality \"production-grade\" software for the mission-critical environment of a <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" target=\"_blank\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> or clinic. Certification is an added challenge in some markets.<sup id=\"rdp-ebb-cite_ref-KaropkaFree14_30-0\" class=\"reference\"><a href=\"#cite_note-KaropkaFree14-30\" rel=\"external_link\">[30]<\/a><\/sup> Despite these challenges, F\/OSS EHR systems have been viewed as excellent options for community health clinics, small practices, and hospitals that are under-capitalized in the United States and overseas.<sup id=\"rdp-ebb-cite_ref-AminpourUtil14_8-1\" class=\"reference\"><a href=\"#cite_note-AminpourUtil14-8\" rel=\"external_link\">[8]<\/a><\/sup>\n<\/p><p>In terms of open-source licenses, the majority of F\/OSS projects use GPL, which allows free modification but requires that any modification be contributed back to that open-source project under the same license. This creates a \"poison pill,\" making it difficult for commercial entities to integrate GPL open-source components into a module that also has proprietary software. This can be viewed as a \"restriction\" with regards to open-source licensing. Restrictiveness of the license also affects a number of contributors and accessibility of the source code.<sup id=\"rdp-ebb-cite_ref-LernerTheScope02_23-2\" class=\"reference\"><a href=\"#cite_note-LernerTheScope02-23\" rel=\"external_link\">[23]<\/a><\/sup>\n<\/p><p>With regards to developers, this study found the average F\/OSS contributor is well-educated, young, and male; the F\/OSS community is male dominant, as women contributors are only 2%. The reasons for gender inequities remain unclear but may involve women facing hybrid discrimination from a F\/OSS community.<sup id=\"rdp-ebb-cite_ref-MahmodWomen10_31-0\" class=\"reference\"><a href=\"#cite_note-MahmodWomen10-31\" rel=\"external_link\">[31]<\/a><\/sup> Our research shows marital status and having kids older than six years old increases the probability a person will volunteer.<sup id=\"rdp-ebb-cite_ref-SelbeePatterns00_32-0\" class=\"reference\"><a href=\"#cite_note-SelbeePatterns00-32\" rel=\"external_link\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HaygheVolunteers91_33-0\" class=\"reference\"><a href=\"#cite_note-HaygheVolunteers91-33\" rel=\"external_link\">[33]<\/a><\/sup>\n<\/p><p>One of the most common aspects of F\/OSS is that they are based on voluntary efforts of developers. The goal of this research was to understand what motivates these developers to contribute to a health care F\/OSS project. Learning new skills and becoming a better programmer is one of the motivations this research found (35%, 34\/96), which is similar to previous findings about mainstream F\/OSS developers (36.5%).<sup id=\"rdp-ebb-cite_ref-AroraFree_28-1\" class=\"reference\"><a href=\"#cite_note-AroraFree-28\" rel=\"external_link\">[28]<\/a><\/sup> Learning and acquiring new skills appears to be another important motivation for many developers.\n<\/p><p>Intrinsic motivations, such as altruism, are high among health care developers; 47.5% (48\/101) of developers worked on the programs for this reason. Only 16% of developers in other research papers stated that altruism was their prime motivator.<sup id=\"rdp-ebb-cite_ref-HarsWorking02_21-1\" class=\"reference\"><a href=\"#cite_note-HarsWorking02-21\" rel=\"external_link\">[21]<\/a><\/sup> Contrary to our research findings, some previous research papers<sup id=\"rdp-ebb-cite_ref-LakhaniWhy05_34-0\" class=\"reference\"><a href=\"#cite_note-LakhaniWhy05-34\" rel=\"external_link\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LuthigerPervasive07_35-0\" class=\"reference\"><a href=\"#cite_note-LuthigerPervasive07-35\" rel=\"external_link\">[35]<\/a><\/sup> found that only a minority were motivated by extrinsic motivation, which involved payments for their participation in F\/OSS projects.\n<\/p><p>One of the unique findings in this research is that 74.2% (75\/101) of health care F\/OSS developers are not health care practitioners, and 45% (45\/99) do not work in the health care field. Being a developer outside of the health care field can be a core problem for the development of usable clinical software with a high degree of functionality. This may serve to explain why open-source EHRs have limited functionality today.\n<\/p><p>Although the open-source projects solve licensing cost problems, there is a need for maintenance and implementation costs, which can be a barrier for organizations without health IT expertise. There are many open-source EHR options on SourceForge, but not all of them can fit in the clinical workflow in the United States or in certain hospitals.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Challenges\">Challenges<\/span><\/h3>\n<p>The availability of motivated developers and the need to continuously improve EHR systems will likely mean F\/OSS EHR will continue to be part of the health care software landscape despite the many challenges these projects face today. Characterizing the challenges and benefits of adopting open-source EHR will be important in understanding the value of these systems.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Usability\">Usability<\/span><\/h4>\n<p>This study shows a potential weakness for F\/OSS EHR projects stemming from developer background and their ability to understand some of the nuances in health care workflows. A large fraction of developers (74.2%, 75\/101) were not health care practitioners, and few developers (54%, 54\/99) had worked in the health care field. A potential solution to this issue would be to have programs that give F\/OSS EHR developers direct access to providers and care venues.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Interoperability\">Interoperability<\/span><\/h4>\n<p>Many hospitals would benefit greatly from integrated software, rather than a disparate group of systems. However, the F\/OSS option can be difficult to interface with commercial systems and may require personnel with multiple skill sets.<sup id=\"rdp-ebb-cite_ref-SchmuhlUseOf13_36-0\" class=\"reference\"><a href=\"#cite_note-SchmuhlUseOf13-36\" rel=\"external_link\">[36]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Privacy_and_security\">Privacy and security<\/span><\/h4>\n<p>Maintaining patient privacy through robust security is a critical aspect of any EHR system. A study of F\/OSS health care systems found the information is often not safeguarded with consent or privacy policies and offers limited protection against unauthorized access or release of information.<sup id=\"rdp-ebb-cite_ref-FloresFunction10_37-0\" class=\"reference\"><a href=\"#cite_note-FloresFunction10-37\" rel=\"external_link\">[37]<\/a><\/sup> However, we did find one open-source EHR system (Tolven eCHR), which supported encryption of health care data at the row level in a relational database. This would make data unreadable even to the database administrator of the system. This is a high degree of security and not typical among EHR systems at the time.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Lack_of_financial_and_professional_expertise\">Lack of financial and professional expertise<\/span><\/h4>\n<p>Low acquisition, installation, and ownership costs make the F\/OSS system an excellent option for organizations with limited capital. However, in general, the health care environments that could benefit the most from F\/OSS EHR systems tend to have low information technology capital budgets and very limited access to <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">health informatics<\/a> professionals. These challenges may prevent the successful implementation of any health IT technology, including F\/OSS EHR.<sup id=\"rdp-ebb-cite_ref-SchmuhlUseOf13_36-1\" class=\"reference\"><a href=\"#cite_note-SchmuhlUseOf13-36\" rel=\"external_link\">[36]<\/a><\/sup> Our study shows that developing countries adopted F\/OSS EHR systems, but the major adoptions were in North America after the HITECH Act. This suggests that even F\/OSS EHR system implementations require substantial financial and workforce resource capacity to succeed.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Cost\">Cost<\/span><\/h4>\n<p>Although F\/OSS software does not have licensing costs, effective implementation still requires skilled staff, time for installation, and time for learning the software. F\/OSS EHR implementation cost can be as high as the proprietary software because of the add-ons, consultation costs, and need for assistance.<sup id=\"rdp-ebb-cite_ref-SchmuhlUseOf13_36-2\" class=\"reference\"><a href=\"#cite_note-SchmuhlUseOf13-36\" rel=\"external_link\">[36]<\/a><\/sup> An important aspect of the total cost of ownership of an EHR system is usability. Poor usability can directly impact the productivity of expensive health care providers and support personnel. The ability to freely modify F\/OSS EHR systems and optimize them for local use likely translates into lower overall cost.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Skilled_information_technology_personnel\">Skilled information technology personnel<\/span><\/h4>\n<p>Adopting F\/OSS EHR requires a large number of IT employees with specific programming skills who understand the program well. A hospital that chooses to use F\/OSS EHR will need to hire developer IT staff and contract with extra IT vendors or consultant support.<sup id=\"rdp-ebb-cite_ref-SchmuhlUseOf13_36-3\" class=\"reference\"><a href=\"#cite_note-SchmuhlUseOf13-36\" rel=\"external_link\">[36]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Limited_functionality_of_clinical_decision_support\">Limited functionality of clinical decision support<\/span><\/h4>\n<p>At an operational level, F\/OSS software also presented reduced functions in decision support and knowledge management. Clinical reasoning, guidelines and protocols, quality assurance, and integrated care were rather limited or nonexistent in most applications.<sup id=\"rdp-ebb-cite_ref-FloresFunction10_37-1\" class=\"reference\"><a href=\"#cite_note-FloresFunction10-37\" rel=\"external_link\">[37]<\/a><\/sup> Developers lack medical knowledge as the majority of them are not health care practitioners, which may affect the efficacy of clinical decision support functionality.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Lack_of_liability_and_accountability\">Lack of liability and accountability<\/span><\/h4>\n<p>One of the major shortcomings is the lack of liability and accountability in F\/OSS. Our study did not address this issue. Few studies of F\/OSS in health care have addressed the risks of using F\/OSS. F\/OSS EHR projects come without warranties regarding the development, release date, or fulfillment of functionality. F\/OSS EHR systems do not have a commercial entity providing support and tend to rely on a volunteer community. This can be a significant risk for a software system that supports the core business of a hospital.<sup id=\"rdp-ebb-cite_ref-SchmuhlUseOf13_36-4\" class=\"reference\"><a href=\"#cite_note-SchmuhlUseOf13-36\" rel=\"external_link\">[36]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Scope_and_limitations\">Scope and limitations<\/span><\/h3>\n<p>Data on F\/OSS EHR projects were primarily collected from SourceForge. Therefore, this study is a snapshot in time, with projects being added and deleted before or after data collection. This amount of \"churn\" reflects a dynamic software category where our findings may not apply in the future.\n<\/p><p>Not all F\/OSS EHR projects were listed on SourceForge; some projects are managed through independent websites. Therefore, some of the projects on SourceForge were outdated, may have contained inaccurate information, and were a small representation of F\/OSS EHR projects as a whole. SourceForge was the main source for most of the data because it tends to be one of the preferred project management platforms for the open source movement and has numerous projects and registered developers.\n<\/p><p>Survey questions were mostly closed-ended which restricted respondents\u2019 answer options. However, we did accommodate all possible answers by providing an additional \"other\" option for some questions. There is a possibility that respondents misunderstood questions as there was no usability testing done on the survey prior to using it in the general F\/OSS EHR community; however, we did not receive any respondent requests to clarify any questions. Furthermore, the survey respondents do not reflect the entire population of open source developers, but we believe it is a reasonable representative of the target population.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>This study highlights a number of important aspects of F\/OSS EHR. Open-source software systems seem to be important to some health care organizations; however, only four F\/OSS EHR systems are ONC-certified in the United States, which creates a barrier to broader use. Health care open-source software also currently lacks directed corporate or governmental support for sustainability and growth of these software programs. We hope this research underlines the challenges that hinder the future of F\/OSS and provides avenues for future research to study and improve adoption of F\/OSS systems in the United States.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h2>\n<p>None declared.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Multimedia_Appendix_1\">Multimedia Appendix 1<\/span><\/h2>\n<p><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/medinform.jmir.org\/article\/downloadSuppFile\/5783\/47862\" target=\"_blank\">Adobe PDF File, 181KB<\/a>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-McDonaldOpen03-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McDonaldOpen03_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McDonald, C.J.; Schadow, G.; Barnes, M. et al. (2003). \"Open source software in medical informatics -- Why, how and what\". <i>International Journal of Medical Informatics<\/i> <b>69<\/b> (2\u20133): 175\u201384. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS1386-5056%2802%2900104-1\" target=\"_blank\">10.1016\/S1386-5056(02)00104-1<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12810121\" target=\"_blank\">12810121<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+source+software+in+medical+informatics+--+Why%2C+how+and+what&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=McDonald%2C+C.J.%3B+Schadow%2C+G.%3B+Barnes%2C+M.+et+al.&rft.au=McDonald%2C+C.J.%3B+Schadow%2C+G.%3B+Barnes%2C+M.+et+al.&rft.date=2003&rft.volume=69&rft.issue=2%E2%80%933&rft.pages=175%E2%80%9384&rft_id=info:doi\/10.1016%2FS1386-5056%2802%2900104-1&rft_id=info:pmid\/12810121&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarnettCOSTAR78-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BarnettCOSTAR78_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barnett, G.O.; Justice, N.S.; Somand, M.E. et al. (1978). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2231744\" target=\"_blank\">\"COSTAR \u2014 A computer-based medical information system for ambulatory care\"<\/a>. <i>Proceedings of the Annual Symposium on Computer Applications in Medical Care<\/i> <b>1978<\/b>: 486\u2013487. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FSCAMC.1978.679955\" target=\"_blank\">10.1109\/SCAMC.1978.679955<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2231744\/\" target=\"_blank\">PMC2231744<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2231744\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2231744<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=COSTAR+%E2%80%94+A+computer-based+medical+information+system+for+ambulatory+care&rft.jtitle=Proceedings+of+the+Annual+Symposium+on+Computer+Applications+in+Medical+Care&rft.aulast=Barnett%2C+G.O.%3B+Justice%2C+N.S.%3B+Somand%2C+M.E.+et+al.&rft.au=Barnett%2C+G.O.%3B+Justice%2C+N.S.%3B+Somand%2C+M.E.+et+al.&rft.date=1978&rft.volume=1978&rft.pages=486%E2%80%93487&rft_id=info:doi\/10.1109%2FSCAMC.1978.679955&rft_id=info:pmc\/PMC2231744&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2231744&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HenryAdoption16-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HenryAdoption16_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Henry, J.W.; Pylypchuk, Y.; Searcy, T. et al. (May 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dashboard.healthit.gov\/evaluations\/data-briefs\/non-federal-acute-care-hospital-ehr-adoption-2008-2015.php\" target=\"_blank\">\"Adoption of Electronic Health Record Systems among U.S. Non-Federal Acute Care Hospitals: 2008-2015\"<\/a>. <i>Health IT Dashboard<\/i>. Office of the National Coordinator for Health Information Technology<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/dashboard.healthit.gov\/evaluations\/data-briefs\/non-federal-acute-care-hospital-ehr-adoption-2008-2015.php\" target=\"_blank\">https:\/\/dashboard.healthit.gov\/evaluations\/data-briefs\/non-federal-acute-care-hospital-ehr-adoption-2008-2015.php<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 September 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Adoption+of+Electronic+Health+Record+Systems+among+U.S.+Non-Federal+Acute+Care+Hospitals%3A+2008-2015&rft.atitle=Health+IT+Dashboard&rft.aulast=Henry%2C+J.W.%3B+Pylypchuk%2C+Y.%3B+Searcy%2C+T.+et+al.&rft.au=Henry%2C+J.W.%3B+Pylypchuk%2C+Y.%3B+Searcy%2C+T.+et+al.&rft.date=May+2016&rft.pub=Office+of+the+National+Coordinator+for+Health+Information+Technology&rft_id=https%3A%2F%2Fdashboard.healthit.gov%2Fevaluations%2Fdata-briefs%2Fnon-federal-acute-care-hospital-ehr-adoption-2008-2015.php&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YeToward03-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-YeToward03_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-YeToward03_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ye, Y.; Kishida, K. (2003). \"Toward an understanding of the motivation open source software developers\". <i>Proceedings of the 25th International Conference on Software Engineering<\/i> <b>2003<\/b>: 419-29. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 076951877X.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+an+understanding+of+the+motivation+open+source+software+developers&rft.jtitle=Proceedings+of+the+25th+International+Conference+on+Software+Engineering&rft.aulast=Ye%2C+Y.%3B+Kishida%2C+K.&rft.au=Ye%2C+Y.%3B+Kishida%2C+K.&rft.date=2003&rft.volume=2003&rft.pages=419-29&rft.isbn=076951877X&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GansMedical05-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GansMedical05_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-GansMedical05_5-1\" rel=\"external_link\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gans, D.; Kralewski, J.; Hammons, T. et al. (2005). \"Medical groups' adoption of electronic health records and information systems\". <i>Health Affairs<\/i> <b>24<\/b> (5): 1323-33. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.24.5.1323\" target=\"_blank\">10.1377\/hlthaff.24.5.1323<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16162580\" target=\"_blank\">16162580<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+groups%27+adoption+of+electronic+health+records+and+information+systems&rft.jtitle=Health+Affairs&rft.aulast=Gans%2C+D.%3B+Kralewski%2C+J.%3B+Hammons%2C+T.+et+al.&rft.au=Gans%2C+D.%3B+Kralewski%2C+J.%3B+Hammons%2C+T.+et+al.&rft.date=2005&rft.volume=24&rft.issue=5&rft.pages=1323-33&rft_id=info:doi\/10.1377%2Fhlthaff.24.5.1323&rft_id=info:pmid\/16162580&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AjamiBarriers13-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AjamiBarriers13_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ajami, S.; Bagheri-Tadi, T. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3766548\" target=\"_blank\">\"Barriers for adopting electronic health records (EHRs) by physicians\"<\/a>. <i>Acta Informatica Medica<\/i> <b>21<\/b> (2): 129\u201334. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5455%2Faim.2013.21.129-134\" target=\"_blank\">10.5455\/aim.2013.21.129-134<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3766548\/\" target=\"_blank\">PMC3766548<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24058254\" target=\"_blank\">24058254<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3766548\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3766548<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Barriers+for+adopting+electronic+health+records+%28EHRs%29+by+physicians&rft.jtitle=Acta+Informatica+Medica&rft.aulast=Ajami%2C+S.%3B+Bagheri-Tadi%2C+T.&rft.au=Ajami%2C+S.%3B+Bagheri-Tadi%2C+T.&rft.date=2013&rft.volume=21&rft.issue=2&rft.pages=129%E2%80%9334&rft_id=info:doi\/10.5455%2Faim.2013.21.129-134&rft_id=info:pmc\/PMC3766548&rft_id=info:pmid\/24058254&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3766548&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FlemingTheFin11-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FlemingTheFin11_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-FlemingTheFin11_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fleming, N.S.; Culler, S.D.; McCorkle, R. et al. (2011). \"The financial and nonfinancial costs of implementing electronic health records in primary care practices\". <i>Health Affairs<\/i> <b>30<\/b> (3): 481-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.2010.0768\" target=\"_blank\">10.1377\/hlthaff.2010.0768<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21383367\" target=\"_blank\">21383367<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+financial+and+nonfinancial+costs+of+implementing+electronic+health+records+in+primary+care+practices&rft.jtitle=Health+Affairs&rft.aulast=Fleming%2C+N.S.%3B+Culler%2C+S.D.%3B+McCorkle%2C+R.+et+al.&rft.au=Fleming%2C+N.S.%3B+Culler%2C+S.D.%3B+McCorkle%2C+R.+et+al.&rft.date=2011&rft.volume=30&rft.issue=3&rft.pages=481-9&rft_id=info:doi\/10.1377%2Fhlthaff.2010.0768&rft_id=info:pmid\/21383367&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AminpourUtil14-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AminpourUtil14_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-AminpourUtil14_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aminpour, F.; Sadoughi, F.; Ahamdi, M. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3963324\" target=\"_blank\">\"Utilization of open source electronic health record around the world: A systematic review\"<\/a>. <i>Journal of Research in Medical Sciences<\/i> <b>19<\/b> (1): 57-64. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3963324\/\" target=\"_blank\">PMC3963324<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24672566\" target=\"_blank\">24672566<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3963324\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3963324<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Utilization+of+open+source+electronic+health+record+around+the+world%3A+A+systematic+review&rft.jtitle=Journal+of+Research+in+Medical+Sciences&rft.aulast=Aminpour%2C+F.%3B+Sadoughi%2C+F.%3B+Ahamdi%2C+M.&rft.au=Aminpour%2C+F.%3B+Sadoughi%2C+F.%3B+Ahamdi%2C+M.&rft.date=2014&rft.volume=19&rft.issue=1&rft.pages=57-64&rft_id=info:pmc\/PMC3963324&rft_id=info:pmid\/24672566&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3963324&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RatwaniElect15-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RatwaniElect15_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ratwani, R.M.; Benda, N.C.; Hettinger, A.Z. et al. (2015). \"Electronic health record vendor adherence to usability certification requirements and testing standards\". <i>JAMA<\/i> <b>314<\/b> (10): 1070-1. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2015.8372\" target=\"_blank\">10.1001\/jama.2015.8372<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26348757\" target=\"_blank\">26348757<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+health+record+vendor+adherence+to+usability+certification+requirements+and+testing+standards&rft.jtitle=JAMA&rft.aulast=Ratwani%2C+R.M.%3B+Benda%2C+N.C.%3B+Hettinger%2C+A.Z.+et+al.&rft.au=Ratwani%2C+R.M.%3B+Benda%2C+N.C.%3B+Hettinger%2C+A.Z.+et+al.&rft.date=2015&rft.volume=314&rft.issue=10&rft.pages=1070-1&rft_id=info:doi\/10.1001%2Fjama.2015.8372&rft_id=info:pmid\/26348757&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BriggsElect14-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BriggsElect14_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Briggs, B.; Carter-Templeton, H. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.himss.org\/electronic-health-record-customization-quality-improvement-project\" target=\"_blank\">\"Electronic health record customization: A quality improvement project\"<\/a>. <i>Online Journal of Nursing Informatics<\/i> <b>18<\/b> (3)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.himss.org\/electronic-health-record-customization-quality-improvement-project\" target=\"_blank\">http:\/\/www.himss.org\/electronic-health-record-customization-quality-improvement-project<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+health+record+customization%3A+A+quality+improvement+project&rft.jtitle=Online+Journal+of+Nursing+Informatics&rft.aulast=Briggs%2C+B.%3B+Carter-Templeton%2C+H.&rft.au=Briggs%2C+B.%3B+Carter-Templeton%2C+H.&rft.date=2014&rft.volume=18&rft.issue=3&rft_id=http%3A%2F%2Fwww.himss.org%2Felectronic-health-record-customization-quality-improvement-project&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-11\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Tate, C. (June 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/amdis.org\/wp-content\/uploads\/2015\/06\/Klas-Review_Tate1.pdf\" target=\"_blank\">\"AMDIS 2015\"<\/a> (PDF). KLAS Enterprises, LLC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/amdis.org\/wp-content\/uploads\/2015\/06\/Klas-Review_Tate1.pdf\" target=\"_blank\">http:\/\/amdis.org\/wp-content\/uploads\/2015\/06\/Klas-Review_Tate1.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 September 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=AMDIS+2015&rft.atitle=&rft.aulast=Tate%2C+C.&rft.au=Tate%2C+C.&rft.date=June+2015&rft.pub=KLAS+Enterprises%2C+LLC&rft_id=http%3A%2F%2Famdis.org%2Fwp-content%2Fuploads%2F2015%2F06%2FKlas-Review_Tate1.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YellowleesStandards08-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YellowleesStandards08_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yellowlees, P.M.; Marks, S.L.; Hogarth, M. et al. (2008). \"Standards-based, open-source electronic health record systems: A desirable future for the U.S. health industry\". <i>Telemedicine Journal and E-Health<\/i> <b>14<\/b> (3): 284-8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Ftmj.2007.0052\" target=\"_blank\">10.1089\/tmj.2007.0052<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18570554\" target=\"_blank\">18570554<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Standards-based%2C+open-source+electronic+health+record+systems%3A+A+desirable+future+for+the+U.S.+health+industry&rft.jtitle=Telemedicine+Journal+and+E-Health&rft.aulast=Yellowlees%2C+P.M.%3B+Marks%2C+S.L.%3B+Hogarth%2C+M.+et+al.&rft.au=Yellowlees%2C+P.M.%3B+Marks%2C+S.L.%3B+Hogarth%2C+M.+et+al.&rft.date=2008&rft.volume=14&rft.issue=3&rft.pages=284-8&rft_id=info:doi\/10.1089%2Ftmj.2007.0052&rft_id=info:pmid\/18570554&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RaymondTheCath199-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RaymondTheCath199_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Raymond, E. (1999). \"The cathedral and the bazaar\". <i>Knowledge, Technology & Policy<\/i> <b>12<\/b> (3): 23\u201349. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs12130-999-1026-0\" target=\"_blank\">10.1007\/s12130-999-1026-0<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+cathedral+and+the+bazaar&rft.jtitle=Knowledge%2C+Technology+%26+Policy&rft.aulast=Raymond%2C+E.&rft.au=Raymond%2C+E.&rft.date=1999&rft.volume=12&rft.issue=3&rft.pages=23%E2%80%9349&rft_id=info:doi\/10.1007%2Fs12130-999-1026-0&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KobayashiOpen12-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KobayashiOpen12_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Kobayashi, S. (2012). \"Open Source Software Development on Medical Domain\". In Kalloniatis, C.. <i>Modern Information Systems<\/i>. InTech. pp. 1\u201316. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5772%2F38117\" target=\"_blank\">10.5772\/38117<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9789535106470.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Open+Source+Software+Development+on+Medical+Domain&rft.atitle=Modern+Information+Systems&rft.aulast=Kobayashi%2C+S.&rft.au=Kobayashi%2C+S.&rft.date=2012&rft.pages=pp.%26nbsp%3B1%E2%80%9316&rft.pub=InTech&rft_id=info:doi\/10.5772%2F38117&rft.isbn=9789535106470&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaglogiannisTowards12-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MaglogiannisTowards12_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-MaglogiannisTowards12_15-1\" rel=\"external_link\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-MaglogiannisTowards12_15-2\" rel=\"external_link\">15.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maglogiannis, I. (2012). \"Towards the adoption of open source and open access electronic health record systems\". <i>Journal of Healthcare Engineering<\/i> <b>3<\/b> (1): 141-161. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1260%2F2040-2295.3.1.141\" target=\"_blank\">10.1260\/2040-2295.3.1.141<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+the+adoption+of+open+source+and+open+access+electronic+health+record+systems&rft.jtitle=Journal+of+Healthcare+Engineering&rft.aulast=Maglogiannis%2C+I.&rft.au=Maglogiannis%2C+I.&rft.date=2012&rft.volume=3&rft.issue=1&rft.pages=141-161&rft_id=info:doi\/10.1260%2F2040-2295.3.1.141&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChelkowskiIneq16-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChelkowskiIneq16_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Che\u0142kowski, T.; Gloor, P.; Jemielniak, D. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4838325\" target=\"_blank\">\"Inequalities in open source software development: analysis of contributor's commits in Apache Software Foundation projects\"<\/a>. <i>PLOS One<\/i> <b>11<\/b> (4): e0152976. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0152976\" target=\"_blank\">10.1371\/journal.pone.0152976<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4838325\/\" target=\"_blank\">PMC4838325<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27096157\" target=\"_blank\">27096157<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4838325\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4838325<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Inequalities+in+open+source+software+development%3A+analysis+of+contributor%27s+commits+in+Apache+Software+Foundation+projects&rft.jtitle=PLOS+One&rft.aulast=Che%C5%82kowski%2C+T.%3B+Gloor%2C+P.%3B+Jemielniak%2C+D.&rft.au=Che%C5%82kowski%2C+T.%3B+Gloor%2C+P.%3B+Jemielniak%2C+D.&rft.date=2016&rft.volume=11&rft.issue=4&rft.pages=e0152976&rft_id=info:doi\/10.1371%2Fjournal.pone.0152976&rft_id=info:pmc\/PMC4838325&rft_id=info:pmid\/27096157&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4838325&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GouldeOpen06-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GouldeOpen06_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Goulde, M.; Brown, E.; Rymer, J.; Hartzband, D. (March 2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.chcf.org\/publications\/2006\/03\/open-source-software-a-primer-for-health-care-leaders\" target=\"_blank\">\"Open Source Software: A Primer for Health Care Leaders\"<\/a>. <i>iHealth Reports<\/i>. Forrester Consulting. pp. 31<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.chcf.org\/publications\/2006\/03\/open-source-software-a-primer-for-health-care-leaders\" target=\"_blank\">http:\/\/www.chcf.org\/publications\/2006\/03\/open-source-software-a-primer-for-health-care-leaders<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 February 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Open+Source+Software%3A+A+Primer+for+Health+Care+Leaders&rft.atitle=iHealth+Reports&rft.aulast=Goulde%2C+M.%3B+Brown%2C+E.%3B+Rymer%2C+J.%3B+Hartzband%2C+D.&rft.au=Goulde%2C+M.%3B+Brown%2C+E.%3B+Rymer%2C+J.%3B+Hartzband%2C+D.&rft.date=March+2006&rft.pages=pp.+31&rft.pub=Forrester+Consulting&rft_id=http%3A%2F%2Fwww.chcf.org%2Fpublications%2F2006%2F03%2Fopen-source-software-a-primer-for-health-care-leaders&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NBVPTheFuture12-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NBVPTheFuture12_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.northbridge.com\/2012-open-source-survey\" target=\"_blank\">\"The Future of Open Source 2012 Survey Results\"<\/a>. North Bridge Venture Partners. 21 May 2012<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.northbridge.com\/2012-open-source-survey\" target=\"_blank\">http:\/\/www.northbridge.com\/2012-open-source-survey<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 September 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Future+of+Open+Source+2012+Survey+Results&rft.atitle=&rft.date=21+May+2012&rft.pub=North+Bridge+Venture+Partners&rft_id=http%3A%2F%2Fwww.northbridge.com%2F2012-open-source-survey&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NBVPTheFuture14-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NBVPTheFuture14_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.northbridge.com\/2014-future-open-source-survey-results-0\" target=\"_blank\">\"The Future of Open Source 2014 Survey Results\"<\/a>. North Bridge Venture Partners. 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.northbridge.com\/2014-future-open-source-survey-results-0\" target=\"_blank\">http:\/\/www.northbridge.com\/2014-future-open-source-survey-results-0<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 September 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Future+of+Open+Source+2014+Survey+Results&rft.atitle=&rft.date=2014&rft.pub=North+Bridge+Venture+Partners&rft_id=http%3A%2F%2Fwww.northbridge.com%2F2014-future-open-source-survey-results-0&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GodfreyEvolution00-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GodfreyEvolution00_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Godfrey, M.W.; Tu, Q. (2000). \"Evolution in open source software: A case study\". <i>Proceedings of the International Conference on Software Maintenance<\/i> <b>2000<\/b>: 131. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FICSM.2000.883030\" target=\"_blank\">10.1109\/ICSM.2000.883030<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 0769507530.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evolution+in+open+source+software%3A+A+case+study&rft.jtitle=Proceedings+of+the+International+Conference+on+Software+Maintenance&rft.aulast=Godfrey%2C+M.W.%3B+Tu%2C+Q.&rft.au=Godfrey%2C+M.W.%3B+Tu%2C+Q.&rft.date=2000&rft.volume=2000&rft.pages=131&rft_id=info:doi\/10.1109%2FICSM.2000.883030&rft.isbn=0769507530&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarsWorking02-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HarsWorking02_21-0\" rel=\"external_link\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-HarsWorking02_21-1\" rel=\"external_link\">21.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hars, A.; Ou, S. (2002). \"Working for free? Motivations for participating in open-source projects\". <i>International Journal of Electronic Commerce<\/i> <b>6<\/b> (3): 25\u201339. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FHICSS.2001.927045\" target=\"_blank\">10.1109\/HICSS.2001.927045<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Working+for+free%3F+Motivations+for+participating+in+open-source+projects&rft.jtitle=International+Journal+of+Electronic+Commerce&rft.aulast=Hars%2C+A.%3B+Ou%2C+S.&rft.au=Hars%2C+A.%3B+Ou%2C+S.&rft.date=2002&rft.volume=6&rft.issue=3&rft.pages=25%E2%80%9339&rft_id=info:doi\/10.1109%2FHICSS.2001.927045&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SenDeterm08-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SenDeterm08_22-0\" rel=\"external_link\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-SenDeterm08_22-1\" rel=\"external_link\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sen, R.; Subramaniam, C.; Nelson, M. (2008). \"Determinants of the choice of open source software license\". <i>Journal of Management Information Systems<\/i> <b>25<\/b> (3): 207\u201340. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2753%2FMIS0742-1222250306\" target=\"_blank\">10.2753\/MIS0742-1222250306<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determinants+of+the+choice+of+open+source+software+license&rft.jtitle=Journal+of+Management+Information+Systems&rft.aulast=Sen%2C+R.%3B+Subramaniam%2C+C.%3B+Nelson%2C+M.&rft.au=Sen%2C+R.%3B+Subramaniam%2C+C.%3B+Nelson%2C+M.&rft.date=2008&rft.volume=25&rft.issue=3&rft.pages=207%E2%80%9340&rft_id=info:doi\/10.2753%2FMIS0742-1222250306&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LernerTheScope02-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LernerTheScope02_23-0\" rel=\"external_link\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-LernerTheScope02_23-1\" rel=\"external_link\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-LernerTheScope02_23-2\" rel=\"external_link\">23.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lerner, J.; Tirole, J. (2002). \"The scope of open source licensing\". <i>Journal of Law, Economics and Organization<\/i> <b>21<\/b> (1): 20\u201356. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3386%2Fw9363\" target=\"_blank\">10.3386\/w9363<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+scope+of+open+source+licensing&rft.jtitle=Journal+of+Law%2C+Economics+and+Organization&rft.aulast=Lerner%2C+J.%3B+Tirole%2C+J.&rft.au=Lerner%2C+J.%3B+Tirole%2C+J.&rft.date=2002&rft.volume=21&rft.issue=1&rft.pages=20%E2%80%9356&rft_id=info:doi\/10.3386%2Fw9363&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JanamanchiTheState09-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JanamanchiTheState09_24-0\" rel=\"external_link\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-JanamanchiTheState09_24-1\" rel=\"external_link\">24.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Janamanchi, B.; Katsamakas, E.; Raghupathi, W. et al. (2009). \"The state and profile of open source software projects in health and medical informatics\". <i>International Journal of Medical Informatics<\/i> <b>78<\/b> (7): 457-72. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2009.02.006\" target=\"_blank\">10.1016\/j.ijmedinf.2009.02.006<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19321384\" target=\"_blank\">19321384<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+state+and+profile+of+open+source+software+projects+in+health+and+medical+informatics&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Janamanchi%2C+B.%3B+Katsamakas%2C+E.%3B+Raghupathi%2C+W.+et+al.&rft.au=Janamanchi%2C+B.%3B+Katsamakas%2C+E.%3B+Raghupathi%2C+W.+et+al.&rft.date=2009&rft.volume=78&rft.issue=7&rft.pages=457-72&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2009.02.006&rft_id=info:pmid\/19321384&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Chengalur-SmithSust10-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Chengalur-SmithSust10_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chengalur-Smith, I.; Sidorova, A.; Daniel, S. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/aisel.aisnet.org\/jais\/vol11\/iss11\/5\/\" target=\"_blank\">\"Sustainability of free\/libre open source projects: A longitudinal study\"<\/a>. <i>Journal of the Association for Information Systems<\/i> <b>11<\/b> (11): 5<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/aisel.aisnet.org\/jais\/vol11\/iss11\/5\/\" target=\"_blank\">http:\/\/aisel.aisnet.org\/jais\/vol11\/iss11\/5\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sustainability+of+free%2Flibre+open+source+projects%3A+A+longitudinal+study&rft.jtitle=Journal+of+the+Association+for+Information+Systems&rft.aulast=Chengalur-Smith%2C+I.%3B+Sidorova%2C+A.%3B+Daniel%2C+S.&rft.au=Chengalur-Smith%2C+I.%3B+Sidorova%2C+A.%3B+Daniel%2C+S.&rft.date=2010&rft.volume=11&rft.issue=11&rft.pages=5&rft_id=http%3A%2F%2Faisel.aisnet.org%2Fjais%2Fvol11%2Fiss11%2F5%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CrowstonOpen02-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CrowstonOpen02_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Crowston, K.; Scozzi, B. (2002). \"Open source software projects as virtual organisations: Competency rallying for software development\". <i>IEE Proceedings - Software<\/i> <b>149<\/b> (1): 3\u201317. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1049%2Fip-sen%3A20020197\" target=\"_blank\">10.1049\/ip-sen:20020197<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+source+software+projects+as+virtual+organisations%3A+Competency+rallying+for+software+development&rft.jtitle=IEE+Proceedings+-+Software&rft.aulast=Crowston%2C+K.%3B+Scozzi%2C+B.&rft.au=Crowston%2C+K.%3B+Scozzi%2C+B.&rft.date=2002&rft.volume=149&rft.issue=1&rft.pages=3%E2%80%9317&rft_id=info:doi\/10.1049%2Fip-sen%3A20020197&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MidhaFactors12-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MidhaFactors12_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Midha, V.; Palvia, P. (2012). \"Factors affecting the success of open source software\". <i>Journal of Systems and Software<\/i> <b>85<\/b> (4): 895-905. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jss.2011.11.010\" target=\"_blank\">10.1016\/j.jss.2011.11.010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Factors+affecting+the+success+of+open+source+software&rft.jtitle=Journal+of+Systems+and+Software&rft.aulast=Midha%2C+V.%3B+Palvia%2C+P.&rft.au=Midha%2C+V.%3B+Palvia%2C+P.&rft.date=2012&rft.volume=85&rft.issue=4&rft.pages=895-905&rft_id=info:doi\/10.1016%2Fj.jss.2011.11.010&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AroraFree-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AroraFree_28-0\" rel=\"external_link\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-AroraFree_28-1\" rel=\"external_link\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Arora, S.; Waterman, A.; David, P.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/web.stanford.edu\/group\/floss-us\/questionnaire\/FLOSS-US-Questionnaire.html\" target=\"_blank\">\"Free\/Libre\/Open Source Software Group - Survey\"<\/a>. Stanford<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/web.stanford.edu\/group\/floss-us\/questionnaire\/FLOSS-US-Questionnaire.html\" target=\"_blank\">http:\/\/web.stanford.edu\/group\/floss-us\/questionnaire\/FLOSS-US-Questionnaire.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 September 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Free%2FLibre%2FOpen+Source+Software+Group+-+Survey&rft.atitle=&rft.aulast=Arora%2C+S.%3B+Waterman%2C+A.%3B+David%2C+P.&rft.au=Arora%2C+S.%3B+Waterman%2C+A.%3B+David%2C+P.&rft.pub=Stanford&rft_id=http%3A%2F%2Fweb.stanford.edu%2Fgroup%2Ffloss-us%2Fquestionnaire%2FFLOSS-US-Questionnaire.html&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HogarthAStudy05-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HogarthAStudy05_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hogarth, M.A.; Turner, S. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1560618\" target=\"_blank\">\"A study of clinically related open source software projects\"<\/a>. <i>AMIA Annual Symposium Proceedings<\/i> <b>2005<\/b>: 330\u20134. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1560618\/\" target=\"_blank\">PMC1560618<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16779056\" target=\"_blank\">16779056<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1560618\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1560618<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+study+of+clinically+related+open+source+software+projects&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Hogarth%2C+M.A.%3B+Turner%2C+S.&rft.au=Hogarth%2C+M.A.%3B+Turner%2C+S.&rft.date=2005&rft.volume=2005&rft.pages=330%E2%80%934&rft_id=info:pmc\/PMC1560618&rft_id=info:pmid\/16779056&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1560618&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KaropkaFree14-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KaropkaFree14_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Karopka, T.; Schmuhl, H.; Demski, H. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3950260\" target=\"_blank\">\"Free\/Libre open source software in health care: A review\"<\/a>. <i>Healthcare Informatics Research<\/i> <b>20<\/b> (1): 11\u201322. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4258%2Fhir.2014.20.1.11\" target=\"_blank\">10.4258\/hir.2014.20.1.11<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3950260\/\" target=\"_blank\">PMC3950260<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24627814\" target=\"_blank\">24627814<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3950260\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3950260<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Free%2FLibre+open+source+software+in+health+care%3A+A+review&rft.jtitle=Healthcare+Informatics+Research&rft.aulast=Karopka%2C+T.%3B+Schmuhl%2C+H.%3B+Demski%2C+H.&rft.au=Karopka%2C+T.%3B+Schmuhl%2C+H.%3B+Demski%2C+H.&rft.date=2014&rft.volume=20&rft.issue=1&rft.pages=11%E2%80%9322&rft_id=info:doi\/10.4258%2Fhir.2014.20.1.11&rft_id=info:pmc\/PMC3950260&rft_id=info:pmid\/24627814&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3950260&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MahmodWomen10-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MahmodWomen10_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mahmod, M.; Yusof, S.; Dahalin, Z. (2010). \"Women contributions to open source software innovation: A social constructivist perspective\". <i>2010 International Symposium on Information Technology<\/i> <b>2010<\/b>: 1433-1438. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FITSIM.2010.5561496\" target=\"_blank\">10.1109\/ITSIM.2010.5561496<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Women+contributions+to+open+source+software+innovation%3A+A+social+constructivist+perspective&rft.jtitle=2010+International+Symposium+on+Information+Technology&rft.aulast=Mahmod%2C+M.%3B+Yusof%2C+S.%3B+Dahalin%2C+Z.&rft.au=Mahmod%2C+M.%3B+Yusof%2C+S.%3B+Dahalin%2C+Z.&rft.date=2010&rft.volume=2010&rft.pages=1433-1438&rft_id=info:doi\/10.1109%2FITSIM.2010.5561496&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SelbeePatterns00-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SelbeePatterns00_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Selbee, L.K.; Reed, P.B. (2000). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www3.carleton.ca\/casr\/Patterns%20of%20Vol.pdf\" target=\"_blank\">\"Patterns of Volunteering Over the Life Cycle\"<\/a> (PDF). Statistics Canada<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www3.carleton.ca\/casr\/Patterns%20of%20Vol.pdf\" target=\"_blank\">http:\/\/www3.carleton.ca\/casr\/Patterns%20of%20Vol.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 September 2016<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Patterns+of+Volunteering+Over+the+Life+Cycle&rft.atitle=&rft.aulast=Selbee%2C+L.K.%3B+Reed%2C+P.B.&rft.au=Selbee%2C+L.K.%3B+Reed%2C+P.B.&rft.date=2000&rft.pub=Statistics+Canada&rft_id=http%3A%2F%2Fwww3.carleton.ca%2Fcasr%2FPatterns%2520of%2520Vol.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HaygheVolunteers91-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HaygheVolunteers91_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hayghe, H.V. (1991). \"Volunteers in the U.S.: Who donates the time?\". <i>Monthly Labor Review<\/i> <b>114<\/b> (2): 17\u201323. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10110196\" target=\"_blank\">10110196<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Volunteers+in+the+U.S.%3A+Who+donates+the+time%3F&rft.jtitle=Monthly+Labor+Review&rft.aulast=Hayghe%2C+H.V.&rft.au=Hayghe%2C+H.V.&rft.date=1991&rft.volume=114&rft.issue=2&rft.pages=17%E2%80%9323&rft_id=info:pmid\/10110196&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LakhaniWhy05-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LakhaniWhy05_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Lakhani, K.R.; Wolf, R. (2005). \"Chapter 1: Why Hackers Do What They Do: Understanding Motivation and Effort in Free\/Open Source Software Projects\". In Feller, J.; Fitzgerald, B.; Hissam, S.; Lakhani, K.R.. <i>Perspectives on Free and Open Source Software<\/i>. MIT Press. pp. 3\u201322. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780262062466.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+1%3A+Why+Hackers+Do+What+They+Do%3A+Understanding+Motivation+and+Effort+in+Free%2FOpen+Source+Software+Projects&rft.atitle=Perspectives+on+Free+and+Open+Source+Software&rft.aulast=Lakhani%2C+K.R.%3B+Wolf%2C+R.&rft.au=Lakhani%2C+K.R.%3B+Wolf%2C+R.&rft.date=2005&rft.pages=pp.%26nbsp%3B3%E2%80%9322&rft.pub=MIT+Press&rft.isbn=9780262062466&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LuthigerPervasive07-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LuthigerPervasive07_35-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Luthiger, B.; Jungwirth, C. (2007). \"Pervasive fun\". <i>First Monday<\/i> <b>12<\/b> (1). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5210%2Ffm.v12i1.1422\" target=\"_blank\">10.5210\/fm.v12i1.1422<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pervasive+fun&rft.jtitle=First+Monday&rft.aulast=Luthiger%2C+B.%3B+Jungwirth%2C+C.&rft.au=Luthiger%2C+B.%3B+Jungwirth%2C+C.&rft.date=2007&rft.volume=12&rft.issue=1&rft_id=info:doi\/10.5210%2Ffm.v12i1.1422&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchmuhlUseOf13-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SchmuhlUseOf13_36-0\" rel=\"external_link\">36.0<\/a><\/sup> <sup><a href=\"#cite_ref-SchmuhlUseOf13_36-1\" rel=\"external_link\">36.1<\/a><\/sup> <sup><a href=\"#cite_ref-SchmuhlUseOf13_36-2\" rel=\"external_link\">36.2<\/a><\/sup> <sup><a href=\"#cite_ref-SchmuhlUseOf13_36-3\" rel=\"external_link\">36.3<\/a><\/sup> <sup><a href=\"#cite_ref-SchmuhlUseOf13_36-4\" rel=\"external_link\">36.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schmuhl, H.; Heinze, O.; Bergh, B. (2013). \"Use of open source software in health care delivery - Results of a qualitative field study\". <i>Yearbook of Medical Informatics<\/i> <b>8<\/b>: 107\u2013113. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23974556\" target=\"_blank\">23974556<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Use+of+open+source+software+in+health+care+delivery+-+Results+of+a+qualitative+field+study&rft.jtitle=Yearbook+of+Medical+Informatics&rft.aulast=Schmuhl%2C+H.%3B+Heinze%2C+O.%3B+Bergh%2C+B.&rft.au=Schmuhl%2C+H.%3B+Heinze%2C+O.%3B+Bergh%2C+B.&rft.date=2013&rft.volume=8&rft.pages=107%E2%80%93113&rft_id=info:pmid\/23974556&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FloresFunction10-37\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FloresFunction10_37-0\" rel=\"external_link\">37.0<\/a><\/sup> <sup><a href=\"#cite_ref-FloresFunction10_37-1\" rel=\"external_link\">37.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Flores Zuniga, A.E.; Win, K.T.; Susilo, W. (2010). \"Functionalities of free and open electronic health record systems\". <i>International Journal of Technology Assessment in Health Care<\/i> <b>26<\/b> (4): 382\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1017%2FS0266462310001121\" target=\"_blank\">10.1017\/S0266462310001121<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20974022\" target=\"_blank\">20974022<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Functionalities+of+free+and+open+electronic+health+record+systems&rft.jtitle=International+Journal+of+Technology+Assessment+in+Health+Care&rft.aulast=Flores+Zuniga%2C+A.E.%3B+Win%2C+K.T.%3B+Susilo%2C+W.&rft.au=Flores+Zuniga%2C+A.E.%3B+Win%2C+K.T.%3B+Susilo%2C+W.&rft.date=2010&rft.volume=26&rft.issue=4&rft.pages=382%E2%80%939&rft_id=info:doi\/10.1017%2FS0266462310001121&rft_id=info:pmid\/20974022&rfr_id=info:sid\/en.wikipedia.org:Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>COSTAR<\/b>: Computer Stored Ambulatory Record\n<\/p><p><b>EHR<\/b>: electronic health record\n<\/p><p><b>F\/OSS<\/b>: free and open source software\n<\/p><p><b>GPL<\/b>: general public license\n<\/p><p><b>HITECH<\/b>: Health Information Technology for Economic and Clinical Health\n<\/p><p><b>IT<\/b>: information technology\n<\/p><p><b>ONC<\/b>: Office of the National Coordinator\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID and DOI was missing and were added to make the references more useful. \n<\/p><p>Per the distribution agreement, the following copyright information is also being added: \n<\/p><p>\u00a9Mona Alsaffar, Peter Yellowlees, Alberto Odor, Michael Hogarth. Originally published in JMIR Medical Informatics (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/medinform.jmir.org\" target=\"_blank\">http:\/\/medinform.jmir.org<\/a>), 24.02.2017.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181453\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.802 seconds\nReal time usage: 0.822 seconds\nPreprocessor visited node count: 28601\/1000000\nPreprocessor generated node count: 38906\/1000000\nPost\u2010expand include size: 210513\/2097152 bytes\nTemplate argument size: 70066\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 792.674 1 - -total\n 86.82% 688.211 1 - Template:Reflist\n 76.32% 605.000 37 - Template:Citation\/core\n 63.11% 500.232 28 - Template:Cite_journal\n 12.61% 99.943 7 - Template:Cite_web\n 7.55% 59.810 1 - Template:Infobox_journal_article\n 7.38% 58.492 46 - Template:Citation\/identifier\n 7.26% 57.567 1 - Template:Infobox\n 5.01% 39.726 2 - Template:Cite_book\n 4.36% 34.590 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:9994-0!*!0!!en!*!* and timestamp 20181214181452 and revision id 29746\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study\">https:\/\/www.limswiki.org\/index.php\/Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","61508b4d2723dec78d6fb5e85ab3f5c1_images":[],"61508b4d2723dec78d6fb5e85ab3f5c1_timestamp":1544811292,"dd7402ef5c2b99b24877d5ee4e9d0720_type":"article","dd7402ef5c2b99b24877d5ee4e9d0720_title":"Intervene: A tool for intersection and visualization of multiple gene or genomic region sets (Khan and Mathelier 2017)","dd7402ef5c2b99b24877d5ee4e9d0720_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets","dd7402ef5c2b99b24877d5ee4e9d0720_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Intervene: A tool for intersection and visualization of multiple gene or genomic region sets\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nIntervene: A tool for intersection and visualization of multiple gene or genomic region setsJournal\n \nBMC BioinformaticsAuthor(s)\n \nKhan, Aziz; Mathelier, AnthonyAuthor affiliation(s)\n \nCentre for Molecular Medicine Norway, Norwegian Radium HospitalPrimary contact\n \nEmail: aziz dot khan at ncmm dot uio dot no and anthony dot mathelier at ncmm dot uio dot noYear published\n \n2017Volume and issue\n \n18Page(s)\n \n287DOI\n \n10.1186\/s12859-017-1708-7ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-017-1708-7Download\n \nhttps:\/\/bmcbioinformatics.biomedcentral.com\/track\/pdf\/10.1186\/s12859-017-1708-7 (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Implementation \n4 Results \n\n4.1 An integrated tool for effective visualization of multiple set intersections \n4.2 Venn diagrams module \n4.3 UpSet plots module \n4.4 Pairwise intersection heat maps module \n4.5 An interactive web application \n4.6 Case study: Highlighting co-binding factors in the MCF-7 cell line \n\n\n5 Discussion \n6 Conclusion \n7 Availability and requirements \n8 Abbreviations \n9 Declarations \n\n9.1 Acknowledgements \n\n9.1.1 Funding \n9.1.2 Availability of data and materials \n9.1.3 Author\u2019s contributions \n9.1.4 Competing interests \n9.1.5 Publisher\u2019s note \n\n\n\n\n10 Additional files \n11 References \n12 Notes \n\n\n\nAbstract \nBackground: A common task for scientists relies on comparing lists of genes or genomic regions derived from high-throughput sequencing experiments. While several tools exist to intersect and visualize sets of genes, similar tools dedicated to the visualization of genomic region sets are currently limited.\nResults: To address this gap, we have developed the Intervene tool, which provides an easy and automated interface for the effective intersection and visualization of genomic region or list sets, thus facilitating their analysis and interpretation. Intervene contains three modules: venn to generate Venn diagrams of up to six sets, upset to generate UpSet plots of multiple sets, and pairwise to compute and visualize intersections of multiple sets as clustered heat maps. Intervene, and its interactive web ShinyApp companion, generate publication-quality figures for the interpretation of genomic region and list sets.\nConclusions: Intervene and its web application companion provide an easy command line and an interactive web interface to compute intersections of multiple genomic and list sets. They have the capacity to plot intersections using easy-to-interpret visual approaches. Intervene is developed and designed to meet the needs of both computer scientists and biologists. The source code is freely available at https:\/\/bitbucket.org\/CBGR\/intervene, with the web application available at https:\/\/asntech.shinyapps.io\/intervene.\nKeywords: visualization, Venn diagrams, UpSet plots, heat maps, genome analysis\n\nBackground \nEffective visualization of transcriptomic, genomic, and epigenomic data generated by next-generation sequencing-based high-throughput assays have become an area of great interest. Most of the data sets generated by such assays are lists of genes or variants, and genomic region sets. The genomic region sets represent genomic locations for specific features, such as transcription factor \u2013 DNA interactions, transcription start sites, histone modifications, and DNase hypersensitivity sites. A common task in the interpretation of these features is to find similarities, differences, and enrichments between such sets, which come from different samples, experimental conditions, or cell and tissue types.\nClassically, the intersection or overlap between different sets, such as gene lists, is represented by Venn diagrams[1] or Edwards-Venn.[2] If the number of sets exceeds four, such diagrams become complex and difficult to interpret. The key challenge is that there are 2n combinations to visually represent when considering n sets. An alternative approach, the UpSet plots, was introduced to depict the intersection of more than three sets.[3] The advantage of UpSet plots is their capacity to rank the intersections and alternatively hide combinations without intersection, which is not possible using a Venn diagram. However, with a large number of sets, UpSet plots become an ineffective way of illustrating set intersections. To visualize a large number of sets, one can represent pairwise intersections using a clustered heat map as suggested by Lex and Gehlenborg.[4]\nThere are several web applications and R packages available to compute intersection and visualization of up to six list sets by using Venn diagrams. Although tools exist to perform genomic region set intersections[5][6][7], there is a limited number of tools available to visualize them.[5][6] To our knowledge no tool exists to generate UpSet plots for genomic region sets. Consequently, there is a great need for integrative tools to compute and visualize intersection of multiple sets of both genomic regions and gene\/list sets.\nTo address this need, we developed Intervene, an easy-to-use command line tool to compute and visualize intersections of genomic regions with Venn diagrams, UpSet plots, or clustered heat maps. Moreover, we provide an interactive web application companion to upload list sets or the output of Intervene to further customize plots. \n\nImplementation \nIntervene comes as a command line tool, along with an interactive Shiny web application to customize the visual representation of intersections. The command line tool is implemented in Python (version 2.7) and R programming language (version 3.3.2). The build also works with Python versions 3.4, 3.5, and 3.6. The accompanying web interface is developed using Shiny (version 1.0.0), a web application framework for R. Intervene uses pybedtools[6] to perform genomic region set intersections and Seaborn (https:\/\/seaborn.pydata.org\/), Matplotlib[7], UpSetR[8], and Corrplot[9] to generate figures. The web application uses the R package Venerable[10] for different types of Venn diagrams, UpSetR for UpSet plots, and heatmap.2 and Corrplot for pairwise intersection clustered heat maps. The UpSet module of the web ShinyApp was derived from the UpSetR[8] ShinyApp, which was extended by adding more options and features to customize the UpSet plots.\nIntervene can be installed by using pip install intervene or using the source code available on bitbucket https:\/\/bitbucket.org\/CBGR\/intervene. The tool has been tested on Linux and MAC systems. The Shiny web application is hosted with shinyapps.io by RStudio, and is compatible with all modern web browsers. A detailed documentation including installation instructions and how to use the tool is provided in Additional file 1 and is available at http:\/\/intervene.readthedocs.io.\n\nResults \nAn integrated tool for effective visualization of multiple set intersections \nAs visualization of sets and their intersections is becoming more and more challenging due to the increasing number of generated data sets, there is a strong need to have an integrated tool to compute and visualize intersections effectively. To address this challenge, we have developed Intervene, which is composed of three different modules, accessible through the subcommands venn, upset, and pairwise. Intervene accepts two types of input files: genomic regions in BED, GFF, or VCF format and gene\/name lists in plain text format. A detailed sketch of Intervene\u2019s command line interface and web application utility with types of inputs is provided in Fig. 1.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. A sketch of Intervene\u2019s command line interface and web application, and input data type\n\n\n\nIntervene provides flexibility to the user to choose figure colors, label text, size, resolution, and type to make them publication-standard quality. To read the help about any module, the user can type intervene < subcommand > \u2212-help on the command line. Furthermore, Intervene produces results as text files, which can be easily imported to the web application for interactive visualization and customization of plots (see \u201cAn interactive web application\u201d section).\n\nVenn diagrams module \nVenn diagrams are the classical approach to show intersections between sets. There are several web-based applications and R packages available to visualize intersections of up-to six list sets in classical Venn, Euler, or Edward\u2019s diagrams.[11][12][13][14][15][16] However, a very limited number of tools are available to visualize genomic region intersections using classical Venn diagrams.[5][6]\nIntervene provides up-to six-way classical Venn diagrams for gene lists or genomic region sets. The associated web interface can also be used to compute the intersection of multiple gene sets and visualize it using different flavors of weighted and unweighted Venn and Euler diagrams. These different types include: classical Venn diagrams (up-to five sets), Chow-Ruskey (up-to five sets), Edwards\u2019 diagrams (up-to five sets), and Battle (up-to nine sets).\nAs an example, one might be interested to calculate the number of overlapping ChIP-seq (chromatin immunoprecipitation followed by sequencing) peaks between different types of histone modification marks (H3K27ac, H3K4me3, and H3K27me3) in human embryonic stem cells (hESC)[17] (Fig. 2a , can be generated with the command intervene venn --test).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Example of Intervene\u2019s command line interface outputs. a A three-way Venn diagram of ChIP-seq peaks of histone modifications (H3K27ac, H3Kme3, and H3K27me3) in hESC obtained from ENCODE[11] b UpSet plot of the intersection of four histone modification peaks in hESC c A heat map of pairwise intersections in terms of Jaccard statistics of super-enhancers in 24 mouse cell and tissue types downloaded from dbSUPER\n\n\n\nUpSet plots module \nWhen the number of sets exceeds four, Venn diagrams become difficult to read and interpret. An alternative and more effective approach is to use UpSet plots to visualize the intersections. An R package with a ShinyApp (https:\/\/gehlenborglab.shinyapps.io\/upsetr\/) and an interactive web-based tool are available at http:\/\/vcg.github.io\/upset to visualize multiple list sets. However, to our knowledge, there is no tool available to draw the UpSet plots for genomic region set intersections. Intervene\u2019s upset subcommand can be used to visualize the intersection of multiple genomic region sets using UpSet plots.\nAs an example, we show the intersections of ChIP-seq peaks for histone modifications (H3K27ac, H3K4me3, H3K27me3, and H3K4me2) in hESC using an UpSet plot, where interactions were ranked by frequency (Fig. 2b, can be generated with the command intervene upset --test). This plot is easier to understand than the four-way Venn diagram (Additional file 1).\n\nPairwise intersection heat maps module \nWith an increasing number of data sets, visualizing all possible intersections becomes unfeasible by using Venn diagrams or UpSet plots. One possibility is to compute pairwise intersections and plot-associated metrics as a clustered heat map. Intervene\u2019s pairwise module provides several metrics to assess intersections, including number of overlaps, fraction of overlap, Jaccard statistics, Fisher\u2019s exact test, and distribution of relative distances. Moreover, the user can choose from different styles of heat maps and clustering approaches.\nAs an example, we obtained the genomic regions of super enhancers in 24 mouse cell type and tissues from dbSUPER[18] and computed the pairwise intersections in terms of Jaccard statistics (Fig. 2c). The triangular heat map shows the pairwise Jaccard index, which is between 0 and 1, where 0 means no overlap and 1 means full overlap. The bar plot shows the number of regions in each cell-type or tissue. This plot can be generated using the command intervene pairwise --test).\n\nAn interactive web application \nIntervene comes with a web application companion to further explore and filter the results in an interactive way. Indeed, intersections between large data sets can be computed locally using Intervene\u2019s command line interface, then the output files can be uploaded to the ShinyApp for further exploration and customization of the figures (Fig. 1 ).\nThe ShinyApp web interface takes four types of inputs: (i) a text\/csv file where each column represents a set, (ii) a binary representation of intersections, (iii) a pairwise matrix of intersections, and (iv) a matrix of overlap counts. The web application provides several easy and intuitive customization options for responsive adjustments of the figures (Figs. 1 and 3). Users can change colors, fonts and plot sizes, change labels, and select and deselect specific sets. These customized and publication-ready figures can be downloaded in PDF, SVG, TIFF, and PNG formats. The pairwise modules also provides three types of correlation coefficients and hierarchical clustering with eight clustering methods and four distance measurement methods. It further provides interactive features to explore data values; this is done by hovering the mouse cursor over each heat map cell, or by using a searchable and sortable data table. The data table can be downloaded as a CSV file and interactive heat maps can be downloaded as HTML. The Shiny-based web application is freely available at https:\/\/asntech.shinyapps.io\/intervene.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Screenshots of web application user interface\n\n\n\nCase study: Highlighting co-binding factors in the MCF-7 cell line \nTranscription factors (TFs) are key proteins regulating transcription through their cooperative binding to the DNA.[19][20] To highlight Intervene\u2019s capabilities, we used the command-line tool and its ShinyApp companion to predict and visualize cooperative interactions between TFs at cis-regulatory regions in the MCF-7 breast cancer cell line. Specifically, we considered (i) TF binding regions derived from uniformly processed TF ChIP-seq experiments compiled in the ReMap database[21] and (ii) promoter and enhancer regions predicted by chromHMM[22] from histone modifications and regulatory factors ChIP-seq.[23] The pairwise module of Intervene was used to compute the fraction of overlap between all pairs of ChIP-seq data sets and regulatory regions. The output matrix was provided to the ShinyApp to compute Spearman correlations of the computed values and to generate the corresponding clustering heat map (default parameters; Fig. 4). \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. MCF-7 cluster heat map. Cluster heat map of the Spearman correlations of fractions of overlap between TF ChIP-seq data sets and regulatory regions in MCF-7. Three clusters (red, green, and blue) are highlighted.\n\n\n\nThe largest cluster (green cluster) was composed of the three key cooperative TFs involved in oestrogen-positive breast cancers: ESR1, FOXA1, and GATA3. They were clustered with enhancer regions where they have been shown to interact.[24] The cluster highlights potential TF cooperators: ARNT, AHR, GREB1, and TLE3. Promoter regions were found in the second largest cluster (red cluster), along with CTCF, STAG1, and RAD21, which are known to orchestrate chromatin architecture in human cells.[25] The last cluster was principally composed by TFAP2C data sets. Taken together, Intervene visually highlighted the cooperation of different sets TFs at MCF-7 promoters and enhancers, in agreement with the literature.\n\nDiscussion \nA comparative analysis of different tools to compute and visualize intersections as Venn diagrams, UpSet plots, and pairwise heat maps is provided in Table 1. Most of the tools available currently can only draw Venn diagrams for up-to six list sets. Intervene provides Venn diagrams, UpSet plots, and pairwise heat maps for both list sets and genomic region sets. To the best of our knowledge, it is the only tool available to draw UpSet plots for the intersections of genomic region sets. Intervene is the first of its kind to allow for the computation and visualization of intersections between multiple genomic region and list sets with three different approaches.\n\n\n\n\n\n\n\nTable 1. Comparison of Intervene with currently available tools to draw Venn diagrams, UpSet plots and pairwise heatmaps\n\n\nApplication\n\nVenn plot types\n\nUpset plot\n\nPairwise heat map\n\nWeighted venn\n\nApplication type\n\nInput type\n\nNumber of inputs\n\nOutput type\n\n\nVennDiagramWeb[12]\n\nClassical Venn, Euler\n\n\u2718\n\n\u2718\n\n\u2718\n\nWeb app\n\nLists\n\n5\n\nTIFF, SVG, PNG, R objects\n\n\nVennPainter[14]\n\nClassical Venn, Edwards, Nested Venn\n\n\u2718\n\n\u2718\n\n\u2718\n\nStand-alone\n\nLists\n\n8\n\nSVG, text\n\n\nVennture[15]\n\nEdwards\n\n\u2718\n\n\u2718\n\n\u2718\n\nStand-alone\n\nLists\n\n8\n\nSVG, text\n\n\nBioVenn[11]\n\nClassical Venn\n\n\u2718\n\n\u2718\n\n\u2713\n\nWeb app\n\nLists\n\n3\n\nSVG, PNG\n\n\njVenn[13]\n\nClassical Venn, Edwards\n\n\u2718\n\n\u2718\n\n\u2718\n\nWeb app\n\nLists\n\n6\n\nSVG, PNG\n\n\nInteractiVenn[16]\n\nEdwards\n\n\u2718\n\n\u2718\n\n\u2718\n\nWeb app\n\nLists\n\n6\n\nSVG, PNG, text\n\n\nUpSetR[3][8]\n\n\u2718\n\n\u2713\n\n\u2718\n\n\u2718\n\nWeb app, R package\n\nLists, binary, counts\n\nMultiple\n\nPDF, PNG\n\n\nChippeakAnno[5]\n\nClassical Venn\n\n\u2718\n\n\u2718\n\n\u2718\n\nR package\n\nGenomic regions\n\n5\n\nPDF, SVG, PNG\n\n\npybedtools[6]\n\nClassical Venn\n\n\u2718\n\nMatrix only\n\n\u2713\n\nCommand line\n\nGenomic regions\n\n3 for Venn, multiple for pairwise\n\nPDF, SVG, PNG\n\n\nIntervene\n\nClassical Venn, Euler, Edwards, Chow-Ruskey, Square, Battle\n\n\u2713\n\n\u2713\n\n\u2713\n\nCommand line, web app\n\nGenomic regions, lists, binary, counts\n\n6 for Venn, multiple for upset and pairwise\n\nPDF, SVG, PNG, TIFF, R objects, text\n\n\n\nIn the near future, Intervene will be integrated to the Galaxy Tool Shed to be easily installed to any Galaxy instance with one click. We plan to develop a dedicated web application allowing users to upload genomic region sets for intersections and visualization.\n\nConclusion \nWe described Intervene as an integrated tool that provides an easy and automated interface for intersection, and effective visualization of genomic region and list sets. To our knowledge, Intervene is the first tool to provide three types of visualization approaches for multiple sets of gene or genomic intervals. The three modules are developed to overcome the situations where the number of sets is large. Intervene and its web application companion are developed and designed to fit the needs of a wide range of scientists.\n\nAvailability and requirements \nProject name: Intervene\nProject home page: https:\/\/bitbucket.org\/CBGR\/intervene\nProject documentation page: http:\/\/intervene.readthedocs.io\nProject Shiny App page: https:\/\/asntech.shinyapps.io\/intervene\/\nOperating system(s): The ShinyApp is platform independent and command line interface is available for Linux and Mac OS X\nProgramming language: Python, R\nOther requirements: Web browser for the ShinyApp\nLicense: GNU GPL\nAny restrictions to use by non-academics: GNU GPL\n\nAbbreviations \nChIP-seq: Chromatin immunoprecipitation followed by sequencing\nENCODE: The Encyclopedia of DNA Elements\nhESCs: Human embryonic stem cells\nSEs: Super-enhancers\nTFs: Transcription factors\n\nDeclarations \nAcknowledgements \nWe thank the developers of the tools we have used to build Intervene and Intervene ShinyApp for sharing their code in open-source software. We thank Marius Gheorghe and Dimitris Polychronopoulos for their useful suggestions and testing the tool, and Annabel Darby for providing suggestions on the manuscript text.\n\nFunding \nThis work has been supported by the Norwegian Research Council, Helse S\u00f8r-\u00d8st, and the University of Oslo through the Centre for Molecular Medicine Norway (NCMM), which is part of the Nordic European Molecular Biology Laboratory Partnership for Molecular Medicine.\n\nAvailability of data and materials \nThe source code of Intervene and test data are freely available at https:\/\/bitbucket.org\/CBGR\/intervene and a detailed documentation can be found at http:\/\/intervene.readthedocs.io. An interactive Shiny App is available at https:\/\/asntech.shinyapps.io\/intervene.\n\nAuthor\u2019s contributions \nAK conceived the project. AK and AM designed the tool. AM supervised the project. AK implemented both Intervene and the Shiny web application. AK wrote the manuscript draft and AM revised it. All authors read and approved the manuscript.\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nPublisher\u2019s note \nSpringer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.\n\nAdditional files \nAdditional File 1: Intervene Documentation - Release v0.5.8: A PDF version of detailed documentation including installation instruction and how to use the command line interface and web application (PDF 1429 kb)\n\nReferences \n\n\n\u2191 Venn, J. (1880). \"On the diagrammatic and mechanical representation of propositions and reasonings\". Philisophical Magazine and Journal of Science 10 (59): 1\u201318. doi:10.1080\/14786448008626877.   \n\n\u2191 Edwards, A.W.F. (2004). Cogwheels of the Mind: The Story of Venn Diagrams. Johns Hopkins University Press. pp. 128. ISBN 9780801874345.   \n\n\u2191 3.0 3.1 Lex, A.; Gehlenborg, N.; Strobelt, H. et al. (2014). \"UpSet: Visualization of Intersecting Sets\". IEEE Transactions on Visualization and Computer Graphics 20 (12): 1983-92. doi:10.1109\/TVCG.2014.2346248.   \n\n\u2191 Lex, A.; Gehlenborg, N. (2014). \"Points of view: Sets and intersections\". IEEE Transactions on Visualization and Computer Graphics 11: 779. doi:10.1038\/nmeth.3033.   \n\n\u2191 5.0 5.1 5.2 5.3 Zhu, L.J.; Gazin, C.; Lawson, N.D. et al. (2010). \"ChIPpeakAnno: A Bioconductor package to annotate ChIP-seq and ChIP-chip data\". BMC Bioinformatics 11: 237. doi:10.1186\/1471-2105-11-237. PMC PMC3098059. PMID 20459804. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3098059 .   \n\n\u2191 6.0 6.1 6.2 6.3 6.4 Dale, R.K.; Pedersen, B.S.; Quinlan, A.R. (2011). \"Pybedtools: A flexible Python library for manipulating genomic datasets and annotations\". Bioinformatics 27 (24): 3423\u20134. doi:10.1093\/bioinformatics\/btr539. PMC PMC3232365. PMID 21949271. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3232365 .   \n\n\u2191 7.0 7.1 Hunter, J.D. (2007). \"Matplotlib: A 2D Graphics Environment\". Computing in Science & Engineering 9 (3). doi:10.1109\/MCSE.2007.55.   \n\n\u2191 8.0 8.1 8.2 Conway, J.R.; Lex, A.; Gehlenborg, N. (25 March 2017). \"UpSetR: An R Package For The Visualization Of Intersecting Sets And Their Properties\". bioRxiv. doi:10.1101\/120600.   \n\n\u2191 Wei, T.; Simko, V. (21 April 2016). \"corrplot: Visualization of a Correlation Matrix\". https:\/\/cran.r-project.org\/package=corrplot .   \n\n\u2191 Swinton, J. (23 September 2009). \"Venn diagrams in R with Vennerable package\". https:\/\/r-forge.r-project.org\/scm\/viewvc.php\/*checkout*\/pkg\/Vennerable\/inst\/doc\/Venn.pdf?revision=58&root=vennerable .   \n\n\u2191 11.0 11.1 11.2 Hulsen, T.; de Vlieg, J.; Alkema, W. (2008). \"BioVenn - A web application for the comparison and visualization of biological lists using area-proportional Venn diagrams\". BMC Genomics 9: 488. doi:10.1186\/1471-2164-9-488. PMC PMC2584113. PMID 18925949. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2584113 .   \n\n\u2191 12.0 12.1 Lam, F.; Lalansingh, C.M.; Babaran, H.E. (2016). \"VennDiagramWeb: A web application for the generation of highly customizable Venn and Euler diagrams\". BMC Bioinformatics 17 (1): 401. doi:10.1186\/s12859-016-1281-5. PMC PMC5048655. PMID 27716034. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5048655 .   \n\n\u2191 13.0 13.1 Bardou, P.; Mariette, J.; Escudi\u00e9, F. et al. (2014). \"jvenn: An interactive Venn diagram viewer\". BMC Bioinformatics 15: 293. doi:10.1186\/1471-2105-15-293. PMC PMC4261873. PMID 25176396. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4261873 .   \n\n\u2191 14.0 14.1 Lin, G.; Chai, J.; Yuan, S. et al. (2016). \"VennPainter: A Tool for the Comparison and Identification of Candidate Genes Based on Venn Diagrams\". PLoS One 11 (4): e0154315. doi:10.1371\/journal.pone.0154315. PMC PMC4847855. PMID 27120465. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4847855 .   \n\n\u2191 15.0 15.1 Martin, B.; Chadwick, W.; Yi, T. et al. (2012). \"VENNTURE--A novel Venn diagram investigational tool for multiple pharmacological dataset analysis\". PLoS One 7 (5): e36911. doi:10.1371\/journal.pone.0036911. PMC PMC3351456. PMID 22606307. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3351456 .   \n\n\u2191 16.0 16.1 Heberle, H.; Meirelles, G.V.; da Silva, F.R. et al. (2015). \"InteractiVenn: A web-based tool for the analysis of sets through Venn diagrams\". BMC Bioinformatics 16: 169. doi:10.1186\/s12859-015-0611-3. PMC PMC4455604. PMID 25994840. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4455604 .   \n\n\u2191 Dunham, I.; Kundaje, A.; Aldred, S.F. et al. (2012). \"An integrated encyclopedia of DNA elements in the human genome\". Nature 489: 7414. doi:10.1038\/nature11247. PMC PMC3439153. PMID 22955616. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3439153 .   \n\n\u2191 Khan, A.; Zhang, X. (2016). \"dbSUPER: a database of super-enhancers in mouse and human genome\". Nucleic Acids Research 44 (D1): D164-71. doi:10.1093\/nar\/gkv1002. PMC PMC4702767. PMID 26438538. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4702767 .   \n\n\u2191 Chronis, C.; Fiziev, P.; Papp, B. et al. (2017). \"Cooperative Binding of Transcription Factors Orchestrates Reprogramming\". Cell 168 (3): 442-459.e20. doi:10.1016\/j.cell.2016.12.016. PMC PMC5302508. PMID 28111071. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5302508 .   \n\n\u2191 Spitz, F.; Furlong, E.E. (2012). \"Transcription factors: From enhancer binding to developmental control\". Nature Reviews Genetics 13 (9): 613-26. doi:10.1038\/nrg3207. PMID 22868264.   \n\n\u2191 Griffon, A.; Barbier, Q.; Dalino, J. et al. (2015). \"Integrative analysis of public ChIP-seq experiments reveals a complex multi-cell regulatory landscape\". Nucleic Acids Research 43 (4): e27. doi:10.1093\/nar\/gku1280.   \n\n\u2191 Ernst, J.; Kellis, M. (2012). \"ChromHMM: automating chromatin-state discovery and characterization\". Nature Methods 9 (3): 215\u20136. doi:10.1038\/nmeth.1906. PMC PMC3577932. PMID 22373907. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3577932 .   \n\n\u2191 Taberlay, P.C.; Statham, A.L.; Kelly, T.K. et al. (2014). \"Reconfiguration of nucleosome-depleted regions at distal regulatory elements accompanies DNA methylation of enhancers and insulators in cancer\". Genome Research 24 (9): 1421-32. doi:10.1101\/gr.163485.113. PMC PMC4158760. PMID 24916973. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4158760 .   \n\n\u2191 Theodorou, V.; Stark, R.; Menon, S.; Carroll, J.S. (2013). \"GATA3 acts upstream of FOXA1 in mediating ESR1 binding by shaping enhancer accessibility\". Genome Research 23 (1): 12-22. doi:10.1101\/gr.139469.112. PMC PMC3530671. PMID 23172872. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3530671 .   \n\n\u2191 Zuin, J.; Dixon, J.R.; van der Reijden, M.I. (2014). \"Cohesin and CTCF differentially affect chromatin architecture and gene expression in human cells\". Proceedings of the National Academy of Sciences of the United States of America 111 (3): 996-1001. doi:10.1073\/pnas.1317788111. PMC PMC3903193. PMID 24335803. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3903193 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\">https:\/\/www.limswiki.org\/index.php\/Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 6 June 2017, at 22:52.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,949 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","dd7402ef5c2b99b24877d5ee4e9d0720_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Intervene_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Intervene: A tool for intersection and visualization of multiple gene or genomic region sets<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: A common task for scientists relies on comparing lists of genes or genomic regions derived from high-throughput sequencing experiments. While several tools exist to intersect and visualize sets of genes, similar tools dedicated to the visualization of genomic region sets are currently limited.\n<\/p><p><b>Results<\/b>: To address this gap, we have developed the Intervene tool, which provides an easy and automated interface for the effective intersection and visualization of genomic region or list sets, thus facilitating their analysis and interpretation. Intervene contains three modules: <i>venn<\/i> to generate Venn diagrams of up to six sets, <i>upset<\/i> to generate UpSet plots of multiple sets, and <i>pairwise<\/i> to compute and visualize intersections of multiple sets as clustered heat maps. Intervene, and its interactive web ShinyApp companion, generate publication-quality figures for the interpretation of genomic region and list sets.\n<\/p><p><b>Conclusions<\/b>: Intervene and its web application companion provide an easy command line and an interactive web interface to compute intersections of multiple genomic and list sets. They have the capacity to plot intersections using easy-to-interpret visual approaches. Intervene is developed and designed to meet the needs of both computer scientists and biologists. The source code is freely available at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/bitbucket.org\/CBGR\/intervene\" target=\"_blank\">https:\/\/bitbucket.org\/CBGR\/intervene<\/a>, with the web application available at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/asntech.shinyapps.io\/intervene\" target=\"_blank\">https:\/\/asntech.shinyapps.io\/intervene<\/a>.\n<\/p><p><b>Keywords<\/b>: visualization, Venn diagrams, UpSet plots, heat maps, genome analysis\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Effective visualization of transcriptomic, genomic, and epigenomic data generated by next-generation sequencing-based high-throughput assays have become an area of great interest. Most of the data sets generated by such assays are lists of genes or variants, and genomic region sets. The genomic region sets represent genomic locations for specific features, such as transcription factor \u2013 DNA interactions, transcription start sites, histone modifications, and DNase hypersensitivity sites. A common task in the interpretation of these features is to find similarities, differences, and enrichments between such sets, which come from different samples, experimental conditions, or cell and tissue types.\n<\/p><p>Classically, the intersection or overlap between different sets, such as gene lists, is represented by Venn diagrams<sup id=\"rdp-ebb-cite_ref-VennOnThe80_1-0\" class=\"reference\"><a href=\"#cite_note-VennOnThe80-1\" rel=\"external_link\">[1]<\/a><\/sup> or Edwards-Venn.<sup id=\"rdp-ebb-cite_ref-EdwardsCogwheels04_2-0\" class=\"reference\"><a href=\"#cite_note-EdwardsCogwheels04-2\" rel=\"external_link\">[2]<\/a><\/sup> If the number of sets exceeds four, such diagrams become complex and difficult to interpret. The key challenge is that there are <i>2<sup>n<\/sup><\/i> combinations to visually represent when considering <i>n<\/i> sets. An alternative approach, the UpSet plots, was introduced to depict the intersection of more than three sets.<sup id=\"rdp-ebb-cite_ref-LexUpSet14_3-0\" class=\"reference\"><a href=\"#cite_note-LexUpSet14-3\" rel=\"external_link\">[3]<\/a><\/sup> The advantage of UpSet plots is their capacity to rank the intersections and alternatively hide combinations without intersection, which is not possible using a Venn diagram. However, with a large number of sets, UpSet plots become an ineffective way of illustrating set intersections. To visualize a large number of sets, one can represent pairwise intersections using a clustered heat map as suggested by Lex and Gehlenborg.<sup id=\"rdp-ebb-cite_ref-LexPoints14_4-0\" class=\"reference\"><a href=\"#cite_note-LexPoints14-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/p><p>There are several web applications and <a href=\"https:\/\/www.limswiki.org\/index.php\/R_(programming_language)\" title=\"R (programming language)\" target=\"_blank\" class=\"wiki-link\" data-key=\"1b0aa598f071aca4c5b4ee08d8bb2bde\">R packages<\/a> available to compute intersection and visualization of up to six list sets by using Venn diagrams. Although tools exist to perform genomic region set intersections<sup id=\"rdp-ebb-cite_ref-ZhuChIP10_5-0\" class=\"reference\"><a href=\"#cite_note-ZhuChIP10-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DalePyebedtools11_6-0\" class=\"reference\"><a href=\"#cite_note-DalePyebedtools11-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HunterMatplotlib07_7-0\" class=\"reference\"><a href=\"#cite_note-HunterMatplotlib07-7\" rel=\"external_link\">[7]<\/a><\/sup>, there is a limited number of tools available to visualize them.<sup id=\"rdp-ebb-cite_ref-ZhuChIP10_5-1\" class=\"reference\"><a href=\"#cite_note-ZhuChIP10-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DalePyebedtools11_6-1\" class=\"reference\"><a href=\"#cite_note-DalePyebedtools11-6\" rel=\"external_link\">[6]<\/a><\/sup> To our knowledge no tool exists to generate UpSet plots for genomic region sets. Consequently, there is a great need for integrative tools to compute and visualize intersection of multiple sets of both genomic regions and gene\/list sets.\n<\/p><p>To address this need, we developed Intervene, an easy-to-use command line tool to compute and visualize intersections of genomic regions with Venn diagrams, UpSet plots, or clustered heat maps. Moreover, we provide an interactive web application companion to upload list sets or the output of Intervene to further customize plots. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<p>Intervene comes as a command line tool, along with an interactive Shiny web application to customize the visual representation of intersections. The command line tool is implemented in Python (version 2.7) and R programming language (version 3.3.2). The build also works with Python versions 3.4, 3.5, and 3.6. The accompanying web interface is developed using Shiny (version 1.0.0), a web application framework for R. Intervene uses pybedtools<sup id=\"rdp-ebb-cite_ref-DalePyebedtools11_6-2\" class=\"reference\"><a href=\"#cite_note-DalePyebedtools11-6\" rel=\"external_link\">[6]<\/a><\/sup> to perform genomic region set intersections and Seaborn (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/seaborn.pydata.org\/\" target=\"_blank\">https:\/\/seaborn.pydata.org\/<\/a>), Matplotlib<sup id=\"rdp-ebb-cite_ref-HunterMatplotlib07_7-1\" class=\"reference\"><a href=\"#cite_note-HunterMatplotlib07-7\" rel=\"external_link\">[7]<\/a><\/sup>, UpSetR<sup id=\"rdp-ebb-cite_ref-ConwayUpSetR17_8-0\" class=\"reference\"><a href=\"#cite_note-ConwayUpSetR17-8\" rel=\"external_link\">[8]<\/a><\/sup>, and Corrplot<sup id=\"rdp-ebb-cite_ref-WeiCorrplot16_9-0\" class=\"reference\"><a href=\"#cite_note-WeiCorrplot16-9\" rel=\"external_link\">[9]<\/a><\/sup> to generate figures. The web application uses the R package Venerable<sup id=\"rdp-ebb-cite_ref-SwintonVenn09_10-0\" class=\"reference\"><a href=\"#cite_note-SwintonVenn09-10\" rel=\"external_link\">[10]<\/a><\/sup> for different types of Venn diagrams, UpSetR for UpSet plots, and heatmap.2 and Corrplot for pairwise intersection clustered heat maps. The UpSet module of the web ShinyApp was derived from the UpSetR<sup id=\"rdp-ebb-cite_ref-ConwayUpSetR17_8-1\" class=\"reference\"><a href=\"#cite_note-ConwayUpSetR17-8\" rel=\"external_link\">[8]<\/a><\/sup> ShinyApp, which was extended by adding more options and features to customize the UpSet plots.\n<\/p><p>Intervene can be installed by using <i>pip install intervene<\/i> or using the source code available on bitbucket <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/bitbucket.org\/CBGR\/intervene\" target=\"_blank\">https:\/\/bitbucket.org\/CBGR\/intervene<\/a>. The tool has been tested on Linux and MAC systems. The Shiny web application is hosted with shinyapps.io by RStudio, and is compatible with all modern web browsers. A detailed documentation including installation instructions and how to use the tool is provided in Additional file 1 and is available at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/intervene.readthedocs.io\" target=\"_blank\">http:\/\/intervene.readthedocs.io<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"An_integrated_tool_for_effective_visualization_of_multiple_set_intersections\">An integrated tool for effective visualization of multiple set intersections<\/span><\/h3>\n<p>As visualization of sets and their intersections is becoming more and more challenging due to the increasing number of generated data sets, there is a strong need to have an integrated tool to compute and visualize intersections effectively. To address this challenge, we have developed Intervene, which is composed of three different modules, accessible through the subcommands <i>venn<\/i>, <i>upset<\/i>, and <i>pairwise<\/i>. Intervene accepts two types of input files: genomic regions in BED, GFF, or VCF format and gene\/name lists in plain text format. A detailed sketch of Intervene\u2019s command line interface and web application utility with types of inputs is provided in Fig. 1.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Khan_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"f27931018353e258703d4bc28622e5d5\"><img alt=\"Fig1 Khan BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/9\/9e\/Fig1_Khan_BMCBioinformatics2017_18.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> A sketch of Intervene\u2019s command line interface and web application, and input data type<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Intervene provides flexibility to the user to choose figure colors, label text, size, resolution, and type to make them publication-standard quality. To read the help about any module, the user can type <i>intervene < subcommand > \u2212-help<\/i> on the command line. Furthermore, Intervene produces results as text files, which can be easily imported to the web application for interactive visualization and customization of plots (see \u201cAn interactive web application\u201d section).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Venn_diagrams_module\">Venn diagrams module<\/span><\/h3>\n<p>Venn diagrams are the classical approach to show intersections between sets. There are several web-based applications and R packages available to visualize intersections of up-to six list sets in classical Venn, Euler, or Edward\u2019s diagrams.<sup id=\"rdp-ebb-cite_ref-HulsenBioVenn08_11-0\" class=\"reference\"><a href=\"#cite_note-HulsenBioVenn08-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LamVennDiagram16_12-0\" class=\"reference\"><a href=\"#cite_note-LamVennDiagram16-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BardouJvenn14_13-0\" class=\"reference\"><a href=\"#cite_note-BardouJvenn14-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LinVennPainter16_14-0\" class=\"reference\"><a href=\"#cite_note-LinVennPainter16-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MartinVENNTURE12_15-0\" class=\"reference\"><a href=\"#cite_note-MartinVENNTURE12-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HeberleInteractiVenn15_16-0\" class=\"reference\"><a href=\"#cite_note-HeberleInteractiVenn15-16\" rel=\"external_link\">[16]<\/a><\/sup> However, a very limited number of tools are available to visualize genomic region intersections using classical Venn diagrams.<sup id=\"rdp-ebb-cite_ref-ZhuChIP10_5-2\" class=\"reference\"><a href=\"#cite_note-ZhuChIP10-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DalePyebedtools11_6-3\" class=\"reference\"><a href=\"#cite_note-DalePyebedtools11-6\" rel=\"external_link\">[6]<\/a><\/sup>\n<\/p><p>Intervene provides up-to six-way classical Venn diagrams for gene lists or genomic region sets. The associated web interface can also be used to compute the intersection of multiple gene sets and visualize it using different flavors of weighted and unweighted Venn and Euler diagrams. These different types include: classical Venn diagrams (up-to five sets), Chow-Ruskey (up-to five sets), Edwards\u2019 diagrams (up-to five sets), and Battle (up-to nine sets).\n<\/p><p>As an example, one might be interested to calculate the number of overlapping ChIP-seq (chromatin immunoprecipitation followed by sequencing) peaks between different types of histone modification marks (H3K27ac, H3K4me3, and H3K27me3) in human embryonic stem cells (hESC)<sup id=\"rdp-ebb-cite_ref-DunhamAnIntegrated12_17-0\" class=\"reference\"><a href=\"#cite_note-DunhamAnIntegrated12-17\" rel=\"external_link\">[17]<\/a><\/sup> (Fig. 2a , can be generated with the command <i>intervene venn --test<\/i>).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Khan_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"3b93c0316f835b1d7dabcdc8ddd16440\"><img alt=\"Fig2 Khan BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/9\/9c\/Fig2_Khan_BMCBioinformatics2017_18.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Example of Intervene\u2019s command line interface outputs. <b>a<\/b> A three-way Venn diagram of ChIP-seq peaks of histone modifications (H3K27ac, H3Kme3, and H3K27me3) in hESC obtained from ENCODE<sup id=\"rdp-ebb-cite_ref-HulsenBioVenn08_11-1\" class=\"reference\"><a href=\"#cite_note-HulsenBioVenn08-11\" rel=\"external_link\">[11]<\/a><\/sup> <b>b<\/b> UpSet plot of the intersection of four histone modification peaks in hESC <b>c<\/b> A heat map of pairwise intersections in terms of Jaccard statistics of super-enhancers in 24 mouse cell and tissue types downloaded from dbSUPER<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"UpSet_plots_module\">UpSet plots module<\/span><\/h3>\n<p>When the number of sets exceeds four, Venn diagrams become difficult to read and interpret. An alternative and more effective approach is to use UpSet plots to visualize the intersections. An R package with a ShinyApp (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/gehlenborglab.shinyapps.io\/upsetr\/\" target=\"_blank\">https:\/\/gehlenborglab.shinyapps.io\/upsetr\/<\/a>) and an interactive web-based tool are available at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/vcg.github.io\/upset\" target=\"_blank\">http:\/\/vcg.github.io\/upset<\/a> to visualize multiple list sets. However, to our knowledge, there is no tool available to draw the UpSet plots for genomic region set intersections. Intervene\u2019s upset subcommand can be used to visualize the intersection of multiple genomic region sets using UpSet plots.\n<\/p><p>As an example, we show the intersections of ChIP-seq peaks for histone modifications (H3K27ac, H3K4me3, H3K27me3, and H3K4me2) in hESC using an UpSet plot, where interactions were ranked by frequency (Fig. 2b, can be generated with the command <i>intervene upset --test<\/i>). This plot is easier to understand than the four-way Venn diagram (Additional file 1).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Pairwise_intersection_heat_maps_module\">Pairwise intersection heat maps module<\/span><\/h3>\n<p>With an increasing number of data sets, visualizing all possible intersections becomes unfeasible by using Venn diagrams or UpSet plots. One possibility is to compute pairwise intersections and plot-associated metrics as a clustered heat map. Intervene\u2019s pairwise module provides several metrics to assess intersections, including number of overlaps, fraction of overlap, Jaccard statistics, Fisher\u2019s exact test, and distribution of relative distances. Moreover, the user can choose from different styles of heat maps and clustering approaches.\n<\/p><p>As an example, we obtained the genomic regions of super enhancers in 24 mouse cell type and tissues from dbSUPER<sup id=\"rdp-ebb-cite_ref-Khan_dbSUPER16_18-0\" class=\"reference\"><a href=\"#cite_note-Khan_dbSUPER16-18\" rel=\"external_link\">[18]<\/a><\/sup> and computed the pairwise intersections in terms of Jaccard statistics (Fig. 2c). The triangular heat map shows the pairwise Jaccard index, which is between 0 and 1, where 0 means no overlap and 1 means full overlap. The bar plot shows the number of regions in each cell-type or tissue. This plot can be generated using the command <i>intervene pairwise --test<\/i>).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"An_interactive_web_application\">An interactive web application<\/span><\/h3>\n<p>Intervene comes with a web application companion to further explore and filter the results in an interactive way. Indeed, intersections between large data sets can be computed locally using Intervene\u2019s command line interface, then the output files can be uploaded to the ShinyApp for further exploration and customization of the figures (Fig. 1 ).\n<\/p><p>The ShinyApp web interface takes four types of inputs: (i) a text\/csv file where each column represents a set, (ii) a binary representation of intersections, (iii) a pairwise matrix of intersections, and (iv) a matrix of overlap counts. The web application provides several easy and intuitive customization options for responsive adjustments of the figures (Figs. 1 and 3). Users can change colors, fonts and plot sizes, change labels, and select and deselect specific sets. These customized and publication-ready figures can be downloaded in PDF, SVG, TIFF, and PNG formats. The pairwise modules also provides three types of correlation coefficients and hierarchical clustering with eight clustering methods and four distance measurement methods. It further provides interactive features to explore data values; this is done by hovering the mouse cursor over each heat map cell, or by using a searchable and sortable data table. The data table can be downloaded as a CSV file and interactive heat maps can be downloaded as HTML. The Shiny-based web application is freely available at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/asntech.shinyapps.io\/intervene\" target=\"_blank\">https:\/\/asntech.shinyapps.io\/intervene<\/a>.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Khan_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"ce07a5894e5ee22242756890ce5814d8\"><img alt=\"Fig3 Khan BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/4\/4b\/Fig3_Khan_BMCBioinformatics2017_18.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Screenshots of web application user interface<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Case_study:_Highlighting_co-binding_factors_in_the_MCF-7_cell_line\">Case study: Highlighting co-binding factors in the MCF-7 cell line<\/span><\/h3>\n<p>Transcription factors (TFs) are key proteins regulating transcription through their cooperative binding to the DNA.<sup id=\"rdp-ebb-cite_ref-ChronisCoop17_19-0\" class=\"reference\"><a href=\"#cite_note-ChronisCoop17-19\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SpitzTranscrip12_20-0\" class=\"reference\"><a href=\"#cite_note-SpitzTranscrip12-20\" rel=\"external_link\">[20]<\/a><\/sup> To highlight Intervene\u2019s capabilities, we used the command-line tool and its ShinyApp companion to predict and visualize cooperative interactions between TFs at cis-regulatory regions in the MCF-7 breast cancer cell line. Specifically, we considered (i) TF binding regions derived from uniformly processed TF ChIP-seq experiments compiled in the ReMap database<sup id=\"rdp-ebb-cite_ref-GriffonIntegrative15_21-0\" class=\"reference\"><a href=\"#cite_note-GriffonIntegrative15-21\" rel=\"external_link\">[21]<\/a><\/sup> and (ii) promoter and enhancer regions predicted by chromHMM<sup id=\"rdp-ebb-cite_ref-ErnstChromHMM12_22-0\" class=\"reference\"><a href=\"#cite_note-ErnstChromHMM12-22\" rel=\"external_link\">[22]<\/a><\/sup> from histone modifications and regulatory factors ChIP-seq.<sup id=\"rdp-ebb-cite_ref-TaberlayReconfig14_23-0\" class=\"reference\"><a href=\"#cite_note-TaberlayReconfig14-23\" rel=\"external_link\">[23]<\/a><\/sup> The pairwise module of Intervene was used to compute the fraction of overlap between all pairs of ChIP-seq data sets and regulatory regions. The output matrix was provided to the ShinyApp to compute Spearman correlations of the computed values and to generate the corresponding clustering heat map (default parameters; Fig. 4). \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Khan_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5666866b95f4d7f2a8a589b816e4ec4d\"><img alt=\"Fig4 Khan BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/2\/29\/Fig4_Khan_BMCBioinformatics2017_18.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> MCF-7 cluster heat map. Cluster heat map of the Spearman correlations of fractions of overlap between TF ChIP-seq data sets and regulatory regions in MCF-7. Three clusters (red, green, and blue) are highlighted.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The largest cluster (green cluster) was composed of the three key cooperative TFs involved in oestrogen-positive breast cancers: ESR1, FOXA1, and GATA3. They were clustered with enhancer regions where they have been shown to interact.<sup id=\"rdp-ebb-cite_ref-TheodorouGATA3_13_24-0\" class=\"reference\"><a href=\"#cite_note-TheodorouGATA3_13-24\" rel=\"external_link\">[24]<\/a><\/sup> The cluster highlights potential TF cooperators: ARNT, AHR, GREB1, and TLE3. Promoter regions were found in the second largest cluster (red cluster), along with CTCF, STAG1, and RAD21, which are known to orchestrate chromatin architecture in human cells.<sup id=\"rdp-ebb-cite_ref-ZuinCohesion14_25-0\" class=\"reference\"><a href=\"#cite_note-ZuinCohesion14-25\" rel=\"external_link\">[25]<\/a><\/sup> The last cluster was principally composed by TFAP2C data sets. Taken together, Intervene visually highlighted the cooperation of different sets TFs at MCF-7 promoters and enhancers, in agreement with the literature.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>A comparative analysis of different tools to compute and visualize intersections as Venn diagrams, UpSet plots, and pairwise heat maps is provided in Table 1. Most of the tools available currently can only draw Venn diagrams for up-to six list sets. Intervene provides Venn diagrams, UpSet plots, and pairwise heat maps for both list sets and genomic region sets. To the best of our knowledge, it is the only tool available to draw UpSet plots for the intersections of genomic region sets. Intervene is the first of its kind to allow for the computation and visualization of intersections between multiple genomic region and list sets with three different approaches.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"9\"><b>Table 1.<\/b> Comparison of Intervene with currently available tools to draw Venn diagrams, UpSet plots and pairwise heatmaps\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Application\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Venn plot types\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Upset plot\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Pairwise heat map\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Weighted venn\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Application type\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Input type\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of inputs\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Output type\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">VennDiagramWeb<sup id=\"rdp-ebb-cite_ref-LamVennDiagram16_12-1\" class=\"reference\"><a href=\"#cite_note-LamVennDiagram16-12\" rel=\"external_link\">[12]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classical Venn, Euler\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Web app\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TIFF, SVG, PNG, R objects\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">VennPainter<sup id=\"rdp-ebb-cite_ref-LinVennPainter16_14-1\" class=\"reference\"><a href=\"#cite_note-LinVennPainter16-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classical Venn, Edwards, Nested Venn\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Stand-alone\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SVG, text\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Vennture<sup id=\"rdp-ebb-cite_ref-MartinVENNTURE12_15-1\" class=\"reference\"><a href=\"#cite_note-MartinVENNTURE12-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Edwards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Stand-alone\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SVG, text\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BioVenn<sup id=\"rdp-ebb-cite_ref-HulsenBioVenn08_11-2\" class=\"reference\"><a href=\"#cite_note-HulsenBioVenn08-11\" rel=\"external_link\">[11]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classical Venn\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Web app\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SVG, PNG\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">jVenn<sup id=\"rdp-ebb-cite_ref-BardouJvenn14_13-1\" class=\"reference\"><a href=\"#cite_note-BardouJvenn14-13\" rel=\"external_link\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classical Venn, Edwards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Web app\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SVG, PNG\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">InteractiVenn<sup id=\"rdp-ebb-cite_ref-HeberleInteractiVenn15_16-1\" class=\"reference\"><a href=\"#cite_note-HeberleInteractiVenn15-16\" rel=\"external_link\">[16]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Edwards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Web app\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SVG, PNG, text\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">UpSetR<sup id=\"rdp-ebb-cite_ref-LexUpSet14_3-1\" class=\"reference\"><a href=\"#cite_note-LexUpSet14-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ConwayUpSetR17_8-2\" class=\"reference\"><a href=\"#cite_note-ConwayUpSetR17-8\" rel=\"external_link\">[8]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Web app, R package\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lists, binary, counts\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDF, PNG\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ChippeakAnno<sup id=\"rdp-ebb-cite_ref-ZhuChIP10_5-3\" class=\"reference\"><a href=\"#cite_note-ZhuChIP10-5\" rel=\"external_link\">[5]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classical Venn\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R package\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Genomic regions\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDF, SVG, PNG\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">pybedtools<sup id=\"rdp-ebb-cite_ref-DalePyebedtools11_6-4\" class=\"reference\"><a href=\"#cite_note-DalePyebedtools11-6\" rel=\"external_link\">[6]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classical Venn\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2718\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Matrix only\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Command line\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Genomic regions\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 for Venn, multiple for pairwise\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDF, SVG, PNG\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Intervene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classical Venn, Euler, Edwards, Chow-Ruskey, Square, Battle\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Command line, web app\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Genomic regions, lists, binary, counts\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6 for Venn, multiple for upset and pairwise\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDF, SVG, PNG, TIFF, R objects, text\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In the near future, Intervene will be integrated to the Galaxy Tool Shed to be easily installed to any <a href=\"https:\/\/www.limswiki.org\/index.php\/Galaxy_(biomedical_software)\" title=\"Galaxy (biomedical software)\" target=\"_blank\" class=\"wiki-link\" data-key=\"ead5d6ebaa8d67744d2f68d454d89ce6\">Galaxy<\/a> instance with one click. We plan to develop a dedicated web application allowing users to upload genomic region sets for intersections and visualization.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>We described Intervene as an integrated tool that provides an easy and automated interface for intersection, and effective visualization of genomic region and list sets. To our knowledge, Intervene is the first tool to provide three types of visualization approaches for multiple sets of gene or genomic intervals. The three modules are developed to overcome the situations where the number of sets is large. Intervene and its web application companion are developed and designed to fit the needs of a wide range of scientists.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Availability_and_requirements\">Availability and requirements<\/span><\/h2>\n<p><b>Project name<\/b>: Intervene\n<\/p><p><b>Project home page<\/b>: <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/bitbucket.org\/CBGR\/intervene\" target=\"_blank\">https:\/\/bitbucket.org\/CBGR\/intervene<\/a>\n<\/p><p><b>Project documentation page<\/b>: <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/intervene.readthedocs.io\" target=\"_blank\">http:\/\/intervene.readthedocs.io<\/a>\n<\/p><p><b>Project Shiny App page<\/b>: <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/asntech.shinyapps.io\/intervene\/\" target=\"_blank\">https:\/\/asntech.shinyapps.io\/intervene\/<\/a>\n<\/p><p><b>Operating system(s)<\/b>: The ShinyApp is platform independent and command line interface is available for Linux and Mac OS X\n<\/p><p><b>Programming language<\/b>: Python, R\n<\/p><p><b>Other requirements<\/b>: Web browser for the ShinyApp\n<\/p><p><b>License<\/b>: GNU GPL\n<\/p><p><b>Any restrictions to use by non-academics<\/b>: GNU GPL\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>ChIP-seq<\/b>: Chromatin immunoprecipitation followed by sequencing\n<\/p><p><b>ENCODE<\/b>: The Encyclopedia of DNA Elements\n<\/p><p><b>hESCs<\/b>: Human embryonic stem cells\n<\/p><p><b>SEs<\/b>: Super-enhancers\n<\/p><p><b>TFs<\/b>: Transcription factors\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Declarations\">Declarations<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h3>\n<p>We thank the developers of the tools we have used to build Intervene and Intervene ShinyApp for sharing their code in open-source software. We thank Marius Gheorghe and Dimitris Polychronopoulos for their useful suggestions and testing the tool, and Annabel Darby for providing suggestions on the manuscript text.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h4>\n<p>This work has been supported by the Norwegian Research Council, Helse S\u00f8r-\u00d8st, and the University of Oslo through the Centre for Molecular Medicine Norway (NCMM), which is part of the Nordic European Molecular Biology Laboratory Partnership for Molecular Medicine.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Availability_of_data_and_materials\">Availability of data and materials<\/span><\/h4>\n<p>The source code of Intervene and test data are freely available at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/bitbucket.org\/CBGR\/intervene\" target=\"_blank\">https:\/\/bitbucket.org\/CBGR\/intervene<\/a> and a detailed documentation can be found at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/intervene.readthedocs.io\" target=\"_blank\">http:\/\/intervene.readthedocs.io<\/a>. An interactive Shiny App is available at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/asntech.shinyapps.io\/intervene\" target=\"_blank\">https:\/\/asntech.shinyapps.io\/intervene<\/a>.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Author.E2.80.99s_contributions\">Author\u2019s contributions<\/span><\/h4>\n<p>AK conceived the project. AK and AM designed the tool. AM supervised the project. AK implemented both Intervene and the Shiny web application. AK wrote the manuscript draft and AM revised it. All authors read and approved the manuscript.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h4>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Publisher.E2.80.99s_note\">Publisher\u2019s note<\/span><\/h4>\n<p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Additional_files\">Additional files<\/span><\/h2>\n<p><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1186%2Fs12859-017-1708-7\/MediaObjects\/12859_2017_1708_MOESM1_ESM.pdf\" target=\"_blank\">Additional File 1<\/a>: Intervene Documentation - Release v0.5.8: A PDF version of detailed documentation including installation instruction and how to use the command line interface and web application (PDF 1429 kb)\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-VennOnThe80-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VennOnThe80_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Venn, J. (1880). \"On the diagrammatic and mechanical representation of propositions and reasonings\". <i>Philisophical Magazine and Journal of Science<\/i> <b>10<\/b> (59): 1\u201318. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F14786448008626877\" target=\"_blank\">10.1080\/14786448008626877<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=On+the+diagrammatic+and+mechanical+representation+of+propositions+and+reasonings&rft.jtitle=Philisophical+Magazine+and+Journal+of+Science&rft.aulast=Venn%2C+J.&rft.au=Venn%2C+J.&rft.date=1880&rft.volume=10&rft.issue=59&rft.pages=1%E2%80%9318&rft_id=info:doi\/10.1080%2F14786448008626877&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EdwardsCogwheels04-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EdwardsCogwheels04_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Edwards, A.W.F. (2004). <i>Cogwheels of the Mind: The Story of Venn Diagrams<\/i>. Johns Hopkins University Press. pp. 128. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780801874345.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Cogwheels+of+the+Mind%3A+The+Story+of+Venn+Diagrams&rft.aulast=Edwards%2C+A.W.F.&rft.au=Edwards%2C+A.W.F.&rft.date=2004&rft.pages=pp.%26nbsp%3B128&rft.pub=Johns+Hopkins+University+Press&rft.isbn=9780801874345&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LexUpSet14-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LexUpSet14_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-LexUpSet14_3-1\" rel=\"external_link\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lex, A.; Gehlenborg, N.; Strobelt, H. et al. (2014). \"UpSet: Visualization of Intersecting Sets\". <i>IEEE Transactions on Visualization and Computer Graphics<\/i> <b>20<\/b> (12): 1983-92. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTVCG.2014.2346248\" target=\"_blank\">10.1109\/TVCG.2014.2346248<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=UpSet%3A+Visualization+of+Intersecting+Sets&rft.jtitle=IEEE+Transactions+on+Visualization+and+Computer+Graphics&rft.aulast=Lex%2C+A.%3B+Gehlenborg%2C+N.%3B+Strobelt%2C+H.+et+al.&rft.au=Lex%2C+A.%3B+Gehlenborg%2C+N.%3B+Strobelt%2C+H.+et+al.&rft.date=2014&rft.volume=20&rft.issue=12&rft.pages=1983-92&rft_id=info:doi\/10.1109%2FTVCG.2014.2346248&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LexPoints14-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LexPoints14_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lex, A.; Gehlenborg, N. (2014). \"Points of view: Sets and intersections\". <i>IEEE Transactions on Visualization and Computer Graphics<\/i> <b>11<\/b>: 779. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnmeth.3033\" target=\"_blank\">10.1038\/nmeth.3033<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Points+of+view%3A+Sets+and+intersections&rft.jtitle=IEEE+Transactions+on+Visualization+and+Computer+Graphics&rft.aulast=Lex%2C+A.%3B+Gehlenborg%2C+N.&rft.au=Lex%2C+A.%3B+Gehlenborg%2C+N.&rft.date=2014&rft.volume=11&rft.pages=779&rft_id=info:doi\/10.1038%2Fnmeth.3033&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhuChIP10-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ZhuChIP10_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-ZhuChIP10_5-1\" rel=\"external_link\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-ZhuChIP10_5-2\" rel=\"external_link\">5.2<\/a><\/sup> <sup><a href=\"#cite_ref-ZhuChIP10_5-3\" rel=\"external_link\">5.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhu, L.J.; Gazin, C.; Lawson, N.D. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3098059\" target=\"_blank\">\"ChIPpeakAnno: A Bioconductor package to annotate ChIP-seq and ChIP-chip data\"<\/a>. <i>BMC Bioinformatics<\/i> <b>11<\/b>: 237. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-11-237\" target=\"_blank\">10.1186\/1471-2105-11-237<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3098059\/\" target=\"_blank\">PMC3098059<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20459804\" target=\"_blank\">20459804<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3098059\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3098059<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ChIPpeakAnno%3A+A+Bioconductor+package+to+annotate+ChIP-seq+and+ChIP-chip+data&rft.jtitle=BMC+Bioinformatics&rft.aulast=Zhu%2C+L.J.%3B+Gazin%2C+C.%3B+Lawson%2C+N.D.+et+al.&rft.au=Zhu%2C+L.J.%3B+Gazin%2C+C.%3B+Lawson%2C+N.D.+et+al.&rft.date=2010&rft.volume=11&rft.pages=237&rft_id=info:doi\/10.1186%2F1471-2105-11-237&rft_id=info:pmc\/PMC3098059&rft_id=info:pmid\/20459804&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3098059&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DalePyebedtools11-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DalePyebedtools11_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-DalePyebedtools11_6-1\" rel=\"external_link\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-DalePyebedtools11_6-2\" rel=\"external_link\">6.2<\/a><\/sup> <sup><a href=\"#cite_ref-DalePyebedtools11_6-3\" rel=\"external_link\">6.3<\/a><\/sup> <sup><a href=\"#cite_ref-DalePyebedtools11_6-4\" rel=\"external_link\">6.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dale, R.K.; Pedersen, B.S.; Quinlan, A.R. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3232365\" target=\"_blank\">\"Pybedtools: A flexible Python library for manipulating genomic datasets and annotations\"<\/a>. <i>Bioinformatics<\/i> <b>27<\/b> (24): 3423\u20134. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtr539\" target=\"_blank\">10.1093\/bioinformatics\/btr539<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3232365\/\" target=\"_blank\">PMC3232365<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21949271\" target=\"_blank\">21949271<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3232365\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3232365<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pybedtools%3A+A+flexible+Python+library+for+manipulating+genomic+datasets+and+annotations&rft.jtitle=Bioinformatics&rft.aulast=Dale%2C+R.K.%3B+Pedersen%2C+B.S.%3B+Quinlan%2C+A.R.&rft.au=Dale%2C+R.K.%3B+Pedersen%2C+B.S.%3B+Quinlan%2C+A.R.&rft.date=2011&rft.volume=27&rft.issue=24&rft.pages=3423%E2%80%934&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtr539&rft_id=info:pmc\/PMC3232365&rft_id=info:pmid\/21949271&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3232365&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HunterMatplotlib07-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HunterMatplotlib07_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-HunterMatplotlib07_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hunter, J.D. (2007). \"Matplotlib: A 2D Graphics Environment\". <i>Computing in Science & Engineering<\/i> <b>9<\/b> (3). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FMCSE.2007.55\" target=\"_blank\">10.1109\/MCSE.2007.55<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Matplotlib%3A+A+2D+Graphics+Environment&rft.jtitle=Computing+in+Science+%26+Engineering&rft.aulast=Hunter%2C+J.D.&rft.au=Hunter%2C+J.D.&rft.date=2007&rft.volume=9&rft.issue=3&rft_id=info:doi\/10.1109%2FMCSE.2007.55&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ConwayUpSetR17-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ConwayUpSetR17_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-ConwayUpSetR17_8-1\" rel=\"external_link\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-ConwayUpSetR17_8-2\" rel=\"external_link\">8.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Conway, J.R.; Lex, A.; Gehlenborg, N. (25 March 2017). \"UpSetR: An R Package For The Visualization Of Intersecting Sets And Their Properties\". <i>bioRxiv<\/i>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2F120600\" target=\"_blank\">10.1101\/120600<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=UpSetR%3A+An+R+Package+For+The+Visualization+Of+Intersecting+Sets+And+Their+Properties&rft.jtitle=bioRxiv&rft.aulast=Conway%2C+J.R.%3B+Lex%2C+A.%3B+Gehlenborg%2C+N.&rft.au=Conway%2C+J.R.%3B+Lex%2C+A.%3B+Gehlenborg%2C+N.&rft.date=25+March+2017&rft_id=info:doi\/10.1101%2F120600&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeiCorrplot16-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WeiCorrplot16_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wei, T.; Simko, V. (21 April 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/cran.r-project.org\/package=corrplot\" target=\"_blank\">\"corrplot: Visualization of a Correlation Matrix\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/cran.r-project.org\/package=corrplot\" target=\"_blank\">https:\/\/cran.r-project.org\/package=corrplot<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=corrplot%3A+Visualization+of+a+Correlation+Matrix&rft.atitle=&rft.aulast=Wei%2C+T.%3B+Simko%2C+V.&rft.au=Wei%2C+T.%3B+Simko%2C+V.&rft.date=21+April+2016&rft_id=https%3A%2F%2Fcran.r-project.org%2Fpackage%3Dcorrplot&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SwintonVenn09-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SwintonVenn09_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Swinton, J. (23 September 2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/r-forge.r-project.org\/scm\/viewvc.php\/*checkout*\/pkg\/Vennerable\/inst\/doc\/Venn.pdf?revision=58&root=vennerable\" target=\"_blank\">\"Venn diagrams in R with Vennerable package\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/r-forge.r-project.org\/scm\/viewvc.php\/*checkout*\/pkg\/Vennerable\/inst\/doc\/Venn.pdf?revision=58&root=vennerable\" target=\"_blank\">https:\/\/r-forge.r-project.org\/scm\/viewvc.php\/*checkout*\/pkg\/Vennerable\/inst\/doc\/Venn.pdf?revision=58&root=vennerable<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Venn+diagrams+in+R+with+Vennerable+package&rft.atitle=&rft.aulast=Swinton%2C+J.&rft.au=Swinton%2C+J.&rft.date=23+September+2009&rft_id=https%3A%2F%2Fr-forge.r-project.org%2Fscm%2Fviewvc.php%2F%2Acheckout%2A%2Fpkg%2FVennerable%2Finst%2Fdoc%2FVenn.pdf%3Frevision%3D58%26root%3Dvennerable&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HulsenBioVenn08-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HulsenBioVenn08_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-HulsenBioVenn08_11-1\" rel=\"external_link\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-HulsenBioVenn08_11-2\" rel=\"external_link\">11.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hulsen, T.; de Vlieg, J.; Alkema, W. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2584113\" target=\"_blank\">\"BioVenn - A web application for the comparison and visualization of biological lists using area-proportional Venn diagrams\"<\/a>. <i>BMC Genomics<\/i> <b>9<\/b>: 488. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2164-9-488\" target=\"_blank\">10.1186\/1471-2164-9-488<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2584113\/\" target=\"_blank\">PMC2584113<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18925949\" target=\"_blank\">18925949<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2584113\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2584113<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioVenn+-+A+web+application+for+the+comparison+and+visualization+of+biological+lists+using+area-proportional+Venn+diagrams&rft.jtitle=BMC+Genomics&rft.aulast=Hulsen%2C+T.%3B+de+Vlieg%2C+J.%3B+Alkema%2C+W.&rft.au=Hulsen%2C+T.%3B+de+Vlieg%2C+J.%3B+Alkema%2C+W.&rft.date=2008&rft.volume=9&rft.pages=488&rft_id=info:doi\/10.1186%2F1471-2164-9-488&rft_id=info:pmc\/PMC2584113&rft_id=info:pmid\/18925949&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2584113&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LamVennDiagram16-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LamVennDiagram16_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-LamVennDiagram16_12-1\" rel=\"external_link\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lam, F.; Lalansingh, C.M.; Babaran, H.E. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5048655\" target=\"_blank\">\"VennDiagramWeb: A web application for the generation of highly customizable Venn and Euler diagrams\"<\/a>. <i>BMC Bioinformatics<\/i> <b>17<\/b> (1): 401. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12859-016-1281-5\" target=\"_blank\">10.1186\/s12859-016-1281-5<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5048655\/\" target=\"_blank\">PMC5048655<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27716034\" target=\"_blank\">27716034<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5048655\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5048655<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=VennDiagramWeb%3A+A+web+application+for+the+generation+of+highly+customizable+Venn+and+Euler+diagrams&rft.jtitle=BMC+Bioinformatics&rft.aulast=Lam%2C+F.%3B+Lalansingh%2C+C.M.%3B+Babaran%2C+H.E.&rft.au=Lam%2C+F.%3B+Lalansingh%2C+C.M.%3B+Babaran%2C+H.E.&rft.date=2016&rft.volume=17&rft.issue=1&rft.pages=401&rft_id=info:doi\/10.1186%2Fs12859-016-1281-5&rft_id=info:pmc\/PMC5048655&rft_id=info:pmid\/27716034&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5048655&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BardouJvenn14-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BardouJvenn14_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-BardouJvenn14_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bardou, P.; Mariette, J.; Escudi\u00e9, F. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4261873\" target=\"_blank\">\"jvenn: An interactive Venn diagram viewer\"<\/a>. <i>BMC Bioinformatics<\/i> <b>15<\/b>: 293. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-15-293\" target=\"_blank\">10.1186\/1471-2105-15-293<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4261873\/\" target=\"_blank\">PMC4261873<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25176396\" target=\"_blank\">25176396<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4261873\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4261873<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=jvenn%3A+An+interactive+Venn+diagram+viewer&rft.jtitle=BMC+Bioinformatics&rft.aulast=Bardou%2C+P.%3B+Mariette%2C+J.%3B+Escudi%C3%A9%2C+F.+et+al.&rft.au=Bardou%2C+P.%3B+Mariette%2C+J.%3B+Escudi%C3%A9%2C+F.+et+al.&rft.date=2014&rft.volume=15&rft.pages=293&rft_id=info:doi\/10.1186%2F1471-2105-15-293&rft_id=info:pmc\/PMC4261873&rft_id=info:pmid\/25176396&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4261873&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LinVennPainter16-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LinVennPainter16_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-LinVennPainter16_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lin, G.; Chai, J.; Yuan, S. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4847855\" target=\"_blank\">\"VennPainter: A Tool for the Comparison and Identification of Candidate Genes Based on Venn Diagrams\"<\/a>. <i>PLoS One<\/i> <b>11<\/b> (4): e0154315. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0154315\" target=\"_blank\">10.1371\/journal.pone.0154315<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4847855\/\" target=\"_blank\">PMC4847855<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27120465\" target=\"_blank\">27120465<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4847855\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4847855<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=VennPainter%3A+A+Tool+for+the+Comparison+and+Identification+of+Candidate+Genes+Based+on+Venn+Diagrams&rft.jtitle=PLoS+One&rft.aulast=Lin%2C+G.%3B+Chai%2C+J.%3B+Yuan%2C+S.+et+al.&rft.au=Lin%2C+G.%3B+Chai%2C+J.%3B+Yuan%2C+S.+et+al.&rft.date=2016&rft.volume=11&rft.issue=4&rft.pages=e0154315&rft_id=info:doi\/10.1371%2Fjournal.pone.0154315&rft_id=info:pmc\/PMC4847855&rft_id=info:pmid\/27120465&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4847855&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MartinVENNTURE12-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MartinVENNTURE12_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-MartinVENNTURE12_15-1\" rel=\"external_link\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Martin, B.; Chadwick, W.; Yi, T. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3351456\" target=\"_blank\">\"VENNTURE--A novel Venn diagram investigational tool for multiple pharmacological dataset analysis\"<\/a>. <i>PLoS One<\/i> <b>7<\/b> (5): e36911. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0036911\" target=\"_blank\">10.1371\/journal.pone.0036911<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3351456\/\" target=\"_blank\">PMC3351456<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22606307\" target=\"_blank\">22606307<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3351456\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3351456<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=VENNTURE--A+novel+Venn+diagram+investigational+tool+for+multiple+pharmacological+dataset+analysis&rft.jtitle=PLoS+One&rft.aulast=Martin%2C+B.%3B+Chadwick%2C+W.%3B+Yi%2C+T.+et+al.&rft.au=Martin%2C+B.%3B+Chadwick%2C+W.%3B+Yi%2C+T.+et+al.&rft.date=2012&rft.volume=7&rft.issue=5&rft.pages=e36911&rft_id=info:doi\/10.1371%2Fjournal.pone.0036911&rft_id=info:pmc\/PMC3351456&rft_id=info:pmid\/22606307&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3351456&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeberleInteractiVenn15-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HeberleInteractiVenn15_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-HeberleInteractiVenn15_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Heberle, H.; Meirelles, G.V.; da Silva, F.R. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4455604\" target=\"_blank\">\"InteractiVenn: A web-based tool for the analysis of sets through Venn diagrams\"<\/a>. <i>BMC Bioinformatics<\/i> <b>16<\/b>: 169. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12859-015-0611-3\" target=\"_blank\">10.1186\/s12859-015-0611-3<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4455604\/\" target=\"_blank\">PMC4455604<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25994840\" target=\"_blank\">25994840<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4455604\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4455604<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=InteractiVenn%3A+A+web-based+tool+for+the+analysis+of+sets+through+Venn+diagrams&rft.jtitle=BMC+Bioinformatics&rft.aulast=Heberle%2C+H.%3B+Meirelles%2C+G.V.%3B+da+Silva%2C+F.R.+et+al.&rft.au=Heberle%2C+H.%3B+Meirelles%2C+G.V.%3B+da+Silva%2C+F.R.+et+al.&rft.date=2015&rft.volume=16&rft.pages=169&rft_id=info:doi\/10.1186%2Fs12859-015-0611-3&rft_id=info:pmc\/PMC4455604&rft_id=info:pmid\/25994840&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4455604&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DunhamAnIntegrated12-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DunhamAnIntegrated12_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dunham, I.; Kundaje, A.; Aldred, S.F. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3439153\" target=\"_blank\">\"An integrated encyclopedia of DNA elements in the human genome\"<\/a>. <i>Nature<\/i> <b>489<\/b>: 7414. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature11247\" target=\"_blank\">10.1038\/nature11247<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3439153\/\" target=\"_blank\">PMC3439153<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22955616\" target=\"_blank\">22955616<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3439153\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3439153<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+integrated+encyclopedia+of+DNA+elements+in+the+human+genome&rft.jtitle=Nature&rft.aulast=Dunham%2C+I.%3B+Kundaje%2C+A.%3B+Aldred%2C+S.F.+et+al.&rft.au=Dunham%2C+I.%3B+Kundaje%2C+A.%3B+Aldred%2C+S.F.+et+al.&rft.date=2012&rft.volume=489&rft.pages=7414&rft_id=info:doi\/10.1038%2Fnature11247&rft_id=info:pmc\/PMC3439153&rft_id=info:pmid\/22955616&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3439153&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Khan_dbSUPER16-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Khan_dbSUPER16_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khan, A.; Zhang, X. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4702767\" target=\"_blank\">\"dbSUPER: a database of super-enhancers in mouse and human genome\"<\/a>. <i>Nucleic Acids Research<\/i> <b>44<\/b> (D1): D164-71. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkv1002\" target=\"_blank\">10.1093\/nar\/gkv1002<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4702767\/\" target=\"_blank\">PMC4702767<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26438538\" target=\"_blank\">26438538<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4702767\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4702767<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=dbSUPER%3A+a+database+of+super-enhancers+in+mouse+and+human+genome&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Khan%2C+A.%3B+Zhang%2C+X.&rft.au=Khan%2C+A.%3B+Zhang%2C+X.&rft.date=2016&rft.volume=44&rft.issue=D1&rft.pages=D164-71&rft_id=info:doi\/10.1093%2Fnar%2Fgkv1002&rft_id=info:pmc\/PMC4702767&rft_id=info:pmid\/26438538&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4702767&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChronisCoop17-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChronisCoop17_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chronis, C.; Fiziev, P.; Papp, B. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5302508\" target=\"_blank\">\"Cooperative Binding of Transcription Factors Orchestrates Reprogramming\"<\/a>. <i>Cell<\/i> <b>168<\/b> (3): 442-459.e20. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cell.2016.12.016\" target=\"_blank\">10.1016\/j.cell.2016.12.016<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5302508\/\" target=\"_blank\">PMC5302508<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28111071\" target=\"_blank\">28111071<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5302508\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5302508<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cooperative+Binding+of+Transcription+Factors+Orchestrates+Reprogramming&rft.jtitle=Cell&rft.aulast=Chronis%2C+C.%3B+Fiziev%2C+P.%3B+Papp%2C+B.+et+al.&rft.au=Chronis%2C+C.%3B+Fiziev%2C+P.%3B+Papp%2C+B.+et+al.&rft.date=2017&rft.volume=168&rft.issue=3&rft.pages=442-459.e20&rft_id=info:doi\/10.1016%2Fj.cell.2016.12.016&rft_id=info:pmc\/PMC5302508&rft_id=info:pmid\/28111071&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5302508&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SpitzTranscrip12-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SpitzTranscrip12_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Spitz, F.; Furlong, E.E. (2012). \"Transcription factors: From enhancer binding to developmental control\". <i>Nature Reviews Genetics<\/i> <b>13<\/b> (9): 613-26. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnrg3207\" target=\"_blank\">10.1038\/nrg3207<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22868264\" target=\"_blank\">22868264<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Transcription+factors%3A+From+enhancer+binding+to+developmental+control&rft.jtitle=Nature+Reviews+Genetics&rft.aulast=Spitz%2C+F.%3B+Furlong%2C+E.E.&rft.au=Spitz%2C+F.%3B+Furlong%2C+E.E.&rft.date=2012&rft.volume=13&rft.issue=9&rft.pages=613-26&rft_id=info:doi\/10.1038%2Fnrg3207&rft_id=info:pmid\/22868264&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GriffonIntegrative15-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GriffonIntegrative15_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Griffon, A.; Barbier, Q.; Dalino, J. et al. (2015). \"Integrative analysis of public ChIP-seq experiments reveals a complex multi-cell regulatory landscape\". <i>Nucleic Acids Research<\/i> <b>43<\/b> (4): e27. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1280\" target=\"_blank\">10.1093\/nar\/gku1280<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integrative+analysis+of+public+ChIP-seq+experiments+reveals+a+complex+multi-cell+regulatory+landscape&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Griffon%2C+A.%3B+Barbier%2C+Q.%3B+Dalino%2C+J.+et+al.&rft.au=Griffon%2C+A.%3B+Barbier%2C+Q.%3B+Dalino%2C+J.+et+al.&rft.date=2015&rft.volume=43&rft.issue=4&rft.pages=e27&rft_id=info:doi\/10.1093%2Fnar%2Fgku1280&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ErnstChromHMM12-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ErnstChromHMM12_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ernst, J.; Kellis, M. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3577932\" target=\"_blank\">\"ChromHMM: automating chromatin-state discovery and characterization\"<\/a>. <i>Nature Methods<\/i> <b>9<\/b> (3): 215\u20136. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnmeth.1906\" target=\"_blank\">10.1038\/nmeth.1906<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3577932\/\" target=\"_blank\">PMC3577932<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22373907\" target=\"_blank\">22373907<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3577932\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3577932<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ChromHMM%3A+automating+chromatin-state+discovery+and+characterization&rft.jtitle=Nature+Methods&rft.aulast=Ernst%2C+J.%3B+Kellis%2C+M.&rft.au=Ernst%2C+J.%3B+Kellis%2C+M.&rft.date=2012&rft.volume=9&rft.issue=3&rft.pages=215%E2%80%936&rft_id=info:doi\/10.1038%2Fnmeth.1906&rft_id=info:pmc\/PMC3577932&rft_id=info:pmid\/22373907&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3577932&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaberlayReconfig14-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaberlayReconfig14_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taberlay, P.C.; Statham, A.L.; Kelly, T.K. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4158760\" target=\"_blank\">\"Reconfiguration of nucleosome-depleted regions at distal regulatory elements accompanies DNA methylation of enhancers and insulators in cancer\"<\/a>. <i>Genome Research<\/i> <b>24<\/b> (9): 1421-32. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.163485.113\" target=\"_blank\">10.1101\/gr.163485.113<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4158760\/\" target=\"_blank\">PMC4158760<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24916973\" target=\"_blank\">24916973<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4158760\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4158760<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reconfiguration+of+nucleosome-depleted+regions+at+distal+regulatory+elements+accompanies+DNA+methylation+of+enhancers+and+insulators+in+cancer&rft.jtitle=Genome+Research&rft.aulast=Taberlay%2C+P.C.%3B+Statham%2C+A.L.%3B+Kelly%2C+T.K.+et+al.&rft.au=Taberlay%2C+P.C.%3B+Statham%2C+A.L.%3B+Kelly%2C+T.K.+et+al.&rft.date=2014&rft.volume=24&rft.issue=9&rft.pages=1421-32&rft_id=info:doi\/10.1101%2Fgr.163485.113&rft_id=info:pmc\/PMC4158760&rft_id=info:pmid\/24916973&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4158760&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TheodorouGATA3_13-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TheodorouGATA3_13_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Theodorou, V.; Stark, R.; Menon, S.; Carroll, J.S. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3530671\" target=\"_blank\">\"GATA3 acts upstream of FOXA1 in mediating ESR1 binding by shaping enhancer accessibility\"<\/a>. <i>Genome Research<\/i> <b>23<\/b> (1): 12-22. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.139469.112\" target=\"_blank\">10.1101\/gr.139469.112<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3530671\/\" target=\"_blank\">PMC3530671<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23172872\" target=\"_blank\">23172872<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3530671\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3530671<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GATA3+acts+upstream+of+FOXA1+in+mediating+ESR1+binding+by+shaping+enhancer+accessibility&rft.jtitle=Genome+Research&rft.aulast=Theodorou%2C+V.%3B+Stark%2C+R.%3B+Menon%2C+S.%3B+Carroll%2C+J.S.&rft.au=Theodorou%2C+V.%3B+Stark%2C+R.%3B+Menon%2C+S.%3B+Carroll%2C+J.S.&rft.date=2013&rft.volume=23&rft.issue=1&rft.pages=12-22&rft_id=info:doi\/10.1101%2Fgr.139469.112&rft_id=info:pmc\/PMC3530671&rft_id=info:pmid\/23172872&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3530671&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZuinCohesion14-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZuinCohesion14_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zuin, J.; Dixon, J.R.; van der Reijden, M.I. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3903193\" target=\"_blank\">\"Cohesin and CTCF differentially affect chromatin architecture and gene expression in human cells\"<\/a>. <i>Proceedings of the National Academy of Sciences of the United States of America<\/i> <b>111<\/b> (3): 996-1001. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1073%2Fpnas.1317788111\" target=\"_blank\">10.1073\/pnas.1317788111<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3903193\/\" target=\"_blank\">PMC3903193<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24335803\" target=\"_blank\">24335803<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3903193\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3903193<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cohesin+and+CTCF+differentially+affect+chromatin+architecture+and+gene+expression+in+human+cells&rft.jtitle=Proceedings+of+the+National+Academy+of+Sciences+of+the+United+States+of+America&rft.aulast=Zuin%2C+J.%3B+Dixon%2C+J.R.%3B+van+der+Reijden%2C+M.I.&rft.au=Zuin%2C+J.%3B+Dixon%2C+J.R.%3B+van+der+Reijden%2C+M.I.&rft.date=2014&rft.volume=111&rft.issue=3&rft.pages=996-1001&rft_id=info:doi\/10.1073%2Fpnas.1317788111&rft_id=info:pmc\/PMC3903193&rft_id=info:pmid\/24335803&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3903193&rfr_id=info:sid\/en.wikipedia.org:Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181452\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.687 seconds\nReal time usage: 0.734 seconds\nPreprocessor visited node count: 21503\/1000000\nPreprocessor generated node count: 36916\/1000000\nPost\u2010expand include size: 186087\/2097152 bytes\nTemplate argument size: 58616\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 676.117 1 - -total\n 84.31% 570.057 1 - Template:Reflist\n 73.16% 494.619 25 - Template:Citation\/core\n 70.14% 474.255 22 - Template:Cite_journal\n 10.69% 72.246 54 - Template:Citation\/identifier\n 9.34% 63.178 1 - Template:Infobox_journal_article\n 9.03% 61.035 1 - Template:Infobox\n 5.48% 37.064 80 - Template:Infobox\/row\n 4.44% 30.045 2 - Template:Cite_web\n 4.01% 27.098 25 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10137-0!*!0!!en!5!* and timestamp 20181214181451 and revision id 30706\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets\">https:\/\/www.limswiki.org\/index.php\/Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","dd7402ef5c2b99b24877d5ee4e9d0720_images":["https:\/\/www.limswiki.org\/images\/9\/9e\/Fig1_Khan_BMCBioinformatics2017_18.gif","https:\/\/www.limswiki.org\/images\/9\/9c\/Fig2_Khan_BMCBioinformatics2017_18.gif","https:\/\/www.limswiki.org\/images\/4\/4b\/Fig3_Khan_BMCBioinformatics2017_18.gif","https:\/\/www.limswiki.org\/images\/2\/29\/Fig4_Khan_BMCBioinformatics2017_18.gif"],"dd7402ef5c2b99b24877d5ee4e9d0720_timestamp":1544811291,"55740ce4aa95ae034b56b161256958e3_type":"article","55740ce4aa95ae034b56b161256958e3_title":"Analyzing the field of bioinformatics with the multi-faceted topic modeling technique (Heo et al. 2017)","55740ce4aa95ae034b56b161256958e3_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique","55740ce4aa95ae034b56b161256958e3_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Analyzing the field of bioinformatics with the multi-faceted topic modeling technique\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nAnalyzing the field of bioinformatics with the multi-faceted topic modeling techniqueJournal\n \nBMC BioinformaticsAuthor(s)\n \nHeo, Go Eun; Kang, Keun Young; Song, Min; Lee, Jeong-HoonAuthor affiliation(s)\n \nYonsei University, POSTECHPrimary contact\n \nEmail: min dot song at yonsei dot ac dot krYear published\n \n2017Volume and issue\n \n18 (Suppl 7)Page(s)\n \n251DOI\n \n10.1186\/s12859-017-1640-xISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-017-1640-xDownload\n \nhttps:\/\/bmcbioinformatics.biomedcentral.com\/track\/pdf\/10.1186\/s12859-017-1640-x (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Related work \n\n3.1 Bibliometric analysis \n3.2 Text mining applied to bibliometrics \n\n\n4 Methods \n\n4.1 Data collection \n4.2 Data preprocessing and keyphrase extraction \n4.3 ACT model application \n4.4 Evaluation \n\n\n5 Results \n\n5.1 Topic analysis per period \n\n5.1.1 First period analysis \n5.1.2 Second period analysis \n5.1.3 Third period analysis \n5.1.4 Fourth period analysis \n\n\n\n\n6 Discussion \n\n6.1 Time series analysis \n6.2 Integrated view of graph pattern analysis \n\n\n7 Conclusions \n8 Declarations \n\n8.1 Acknowledgements \n\n8.1.1 Funding \n8.1.2 Availability of data and materials \n8.1.3 Authors' contributions \n8.1.4 Competing interests \n8.1.5 Consent for publication \n8.1.6 Ethics approval and consent to participate \n8.1.7 About this supplement \n8.1.8 Publisher\u2019s note \n\n\n\n\n9 Additional files \n10 References \n11 Notes \n\n\n\nAbstract \nBackground: Bioinformatics is an interdisciplinary field at the intersection of molecular biology and computing technology. To characterize the field as a convergent domain, researchers have used bibliometrics, augmented with text-mining techniques for content analysis. In previous studies, Latent Dirichlet Allocation (LDA) was the most representative topic modeling technique for identifying topic structure of subject areas. However, as opposed to revealing the topic structure in relation to metadata such as authors, publication date, and journals, LDA only displays the simple topic structure.\nMethods: In this paper, we adopt the Author-Conference-Topic (ACT) model of Tang et al. to study the field of bioinformatics from the perspective of keyphrases, authors, and journals. The ACT model is capable of incorporating the paper, author, and conference into the topic distribution simultaneously. To obtain more meaningful results, we used journals and keyphrases instead of conferences and the bag-of-words. For analysis, we used PubMed to collect forty-six bioinformatics journals from the MEDLINE database. We conducted time series topic analysis over four periods from 1996 to 2015 to further examine the interdisciplinary nature of bioinformatics.\nResults: We analyzed the ACT Model results in each period. Additionally, for further integrated analysis, we conducted a time series analysis among the top-ranked keyphrases, journals, and authors according to their frequency. We also examined the patterns in the top journals by simultaneously identifying the topical probability in each period, as well as the top authors and keyphrases. The results indicate that in recent years diversified topics have become more prevalent, and convergent topics have become more clearly represented.\nConclusion: The results of our analysis imply that over time the field of bioinformatics becomes more interdisciplinary where there is a steady increase in peripheral fields such as conceptual, mathematical, and system biology. These results are confirmed by integrated analysis of topic distribution as well as top ranked keyphrases, authors, and journals.\nKeywords: bioinformatics, text mining, topic modeling, ACT model, keyphrase extraction\n\nBackground \nOver the years, academic subject areas have converged to form a variety of new, interdisciplinary fields. Bioinformatics is one example. Research domains from molecular biology to machine learning are used in conjunction to better understand complex biological systems such as cells, tissues, and the human body. Due to the complexity and broadness of the field, bibliometric analysis is often adopted to assess the current knowledge structure of a subject area, specify the current research themes, and identify the core literature of that area.[1]\nBibliometrics identifies research trends using quantitative measures such as a researcher\u2019s number of publications and citations, journal impact factors, and other indices that can measure impact or productivity of author or journal.[2][3][4][5] In addition, other factors such as the affiliation of authors, collaborations, and citation data are often incorporated into bibliometric analysis.[6][7][8][9]\nPrevious studies mainly rely on quantitative measures and suffer from the lack of content analysis. To incorporate content analysis into bibliometrics, text-mining techniques are applied. Topic-modeling techniques are mostly adopted to identify the topics of a subject area while analyzing that area more abundantly.[10][11][12][13] These techniques allow for enriched content analysis. As an extension of Latent Dirichlet Allocation (LDA), which is the best received topic-modeling technique, Steyvers et al.[14] proposed the author-topic modeling technique that analyzes authors and topics simultaneously. They identify the authors\u2019 impact or productivity of researchers in a given subject area.[15][16] By adding multiple conditions to LDA, Tang et al.[17] suggested a new methodology, called the Author-Conference-Topic (ACT) model that analyzes the author, conference, and topic in one model to understand the subject area in an integrated manner.\nIn this paper, we apply the ACT model to examine the interdisciplinary nature of bioinformatics. Unlike studies that use extended versions of LDA for topic analysis, the ACT model enables us to analyze topic, author, and journal at one time, providing an integrated view for understanding bioinformatics. The research questions that we are to investigate in this paper are: 1) What are the topical trends of bioinformatics over time? 2) Who are the key contributors in major topics of bioinformatics?, and 3) Which journal is leading which topic?\nTo address these questions, we collect PubMed articles in XML format and extract metadata and content such as the PMID, author, year, journal, title, and abstract. From the title and abstract, we extract keyphrases, which provide more meaningful interpretations than single words, as an input of the ACT model. We also divide the collected datasets into four time periods to examine the topic changes over time. The results of ACT model\u2013based analysis show that various topics begin to appear and mixed subject topics become more apparent over time.\nThe rest of the paper is organized as follows. In the \"Related work\" section, we discuss work related to bibliometric analysis and topic modeling. We then describe the proposed method in the \"Methods\" section. We analyze and discuss the results of leading topics, authors, and journals in the \"Results\" and \"Discussion\" sections. Finally, we conclude the paper and suggest future lines of inquiry in the \"Conclusion\" section.\n\nRelated work \nBibliometric analysis \nBibliometric analysis identifies the research trends in a given subject area and core journals or documents, and it helps with contrasting analysis. Many bibliometric studies use the number of published articles or journal impact factors to measure research productivity or to identify core journals in a specific field. Soteriades and Falalgas[3] applied quantitative and qualitative measurements to analyze the fields of preventive medicine, occupational and environmental medicine, epidemiology, and public health using the number of articles and impact factor. Ugolini et al.[4] measured research productivity and evaluated the publication trends in the field of cancer molecular epidemiology. To quantify productivity, they used the number of articles and the average and sum of impact factors. To evaluate publication trends, they collected and divided the keywords from MeSH terms about the publication into six groups. Ramos et al.[18] measured the national research activity of the tuberculosis field, using impact factor and the first author\u2019s address. Claude et al.[19] examined research productivity by using distribution of publications related to medicine and ANN, the subfield of biology. They used the number of publications, impact factor, and journal category compared with national gross domestic product (GDP). In the bioinformatics field, Patra and Mishra[20] used the number of articles, publication of each journal, publication type, and the impact factor of journals to understand the growth of bioinformatics. They also found the core journals in the bioinformatics fields. Using author affiliation, they applied Lotka\u2019s law to assess the distribution of each author\u2019s productivity. Chen et al.[2] identified research trends using statistical methods based on the type of publication, language, and distribution of nation or institution. They measured h-index, adding statistical materials with the number of citations. Through this, they analyzed the research productivity by topic, institution, and journal. In addition, they conducted a keyword analysis to comprehend the research trend in a macroscopic view.\nMainstream bibliometrics research focuses on identifying the knowledge structure of a certain field with quantitative measures. In addition, some studies use author information or the collaboration pattern among authors to understand the certain field. Seglen and Aksnes[9] used the size and the productivity of research groups in the microbiology field in Norway as a measurement for bibliometric analysis. Geaney et al.[7] performed bibliometric analysis and density-equalizing mapping on scientific publications related to type 2 diabetes mellitus. They collected citation data and used various citation-oriented measures such as the number of citations, the average number of citations per journal, the total number of publications, impact factor, and Eigenfactor score. To conduct content analysis and study the collaboration pattern between authors and the core sub-field of AIDS, Mac\u00edas-Chapula and Mijangos-Nolasco[8] analyzed MeSH thesaurus using check tags, main headings, and subheadings of each MeSH term hierarchy. In addition, to measure the national research productivity, they used the authors\u2019 address information. Bornmann and Mutz[6] recently identified the development of modern science by bibliometric analysis. They divide the data into three time periods to analyze the changes of fields over time.\n\nText mining applied to bibliometrics \nRecently, there have been many attempts to apply text-mining techniques to bibliometric analysis to identify the knowledge structure of the field or measure its influence on other researchers and their fields and productivity. Song and Kim[11] collected full-text articles from PubMed Central and computed their citation relation. They infer the knowledge structure and understand the trend of the bioinformatics field. In a similar vein, Song et al.[12] measured the influence and productivity of bioinformatics by mining full-text articles retrieved from PubMed Central. To calculate the field\u2019s productivity, they identified the most productive author, nation, institution, and topic word; to calculate its influence, they identified the most-cited paper, author, and rising researcher. Song et al.[21] analyzed topic evolution in the bioinformatics field using DBLP data in the field of computer science. To identify topic trends over time, they divided a dozen years (2000\u20132011) into four periods and applied Markov Random Field-based topic clustering. For automatic clustering labeling, they calculated topic similarity based on Within-Period Cluster Similarity (WPCS) and Between-Period Cluster Similarity (BPCS). Their approach created topic graphs that show interaction among topics over some period of time. Lee et al.[22] mapped the Alzheimer\u2019s disease field into three different perspectives: indexer, author, and citer. They applied entity-metrics[23] the extended notion of bibliometrics, to analyze the field by constructing four kinds of networks that convey these three perspectives.\nThese studies identify the knowledge structure of a certain field by constructing bibliometric networks or databases with text-mining techniques. The most prevalent approach is to apply topic modeling to content analysis as a part of bibliometrics. Starting from the probabilistic Latent Semantic Indexing (pLSI)[24] model, Latent Dirichlet Allocation (LDA)[25] is the most accepted topic modeling technique for bibliometrics. While each document consists of a set of topics in pLSI, using the LDA model results in a more precise manipulation being added to organize the topics. Yan[13] used the LDA model to measure the influence and popularity of library and information science. He also identified the most-cited area and the patterns in this field. Jeong and Song\u2019s[10] research measured the time gap among three different resources \u2014 web, patent, and scientific publication \u2014 in two research domains by applying the LDA model. The basic input unit for LDA is a set of documents. To organize author information into topics, Rosen-Zvi et al.[16] and Steyvers et al.[14] proposed the author-topic model with different theoretical background. Li et al.[15] identified the relations between authors and topics by using the author-topic model. They analyzed the topic distribution to examine how many authors are associated with a certain topic. Also through the number of authors, they identified topics that are studied by many researchers. Tang et al.[17] proposed the ACT model which identifies paper, author, and conference simultaneously. Additionally, they developed the ArnetMiner system for mining academic research social networks. Tang et al.[26] also supplement ArnetMiner for a topic level expertise search over heterogeneous networks using the ACT model. It generates the most issued topics, author\u2019s interestedness, paper search, academic suggestion, and experts in a specific field. Kim et al.[27] adopted the ACT model in terms of citation analysis. They collected their dataset in the field of oncology from PubMed Central, which provides the full-text articles in the biomedical field. They utilized the ACT model for analyzing citation sentences and journals instead of abstracts and conferences.\nIn conclusion, most previous studies identified knowledge structures by adopting not only bibliometric analysis but text-mining techniques such as the LDA model. To supplement bibliometric analysis, there are many attempts to incorporate content analysis into bibliometrics by adopting the LDA model text-mining techniques. However, the main limitation of this application of the LDA model, the representative method for trend analysis, is that it only explains topical trends by using one parameter such as the bag-of-words on documents via topical terms. It is not sufficient to conduct comprehensive analysis for understanding knowledge disciplines. Therefore, in this paper, we apply the ACT model to the bioinformatics field for integrated analysis. Applying the ACT model, we aim to explore the importance of authors and journals in relation to topics. We divided the collected datasets into four periods to trace the changes of topic, author, and journal ranking over time, combining the results with bibliometric analysis.\n\nMethods \nIn this section, we describe data collection, preprocessing, and keyphrase extraction to feed input into the ACT model. Figure 1 illustrates the overflow of our approach; detailed descriptions of each component are provided in the following section.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Research overflow. Research overflow of our approach consists of data collection, preprocessing, keyphrase extraction, ACT model application, and topic analysis.\n\n\n\nData collection \nFor analysis, we collected the 48 journals in the bioinformatics field as used by Song and Kim.[11] Forty-six out of the 48 journals were found via the advanced search tool provided by PubMed. Two journals, Advanced Bioinformatics and Genome Integration, were not retrieved from PubMed. We downloaded the 46 PubMed-listed journals in XML format (Table 1). The total number of papers indexed in these journals was 241,569; Biochemistry had the greatest number of papers with 62,270, accounting for 25.78% of the collected publications.\n\n\n\n\n\n\n\nTable 1. Statistics of collected publications\n\n\nRanking\n\nJournal name\n\nNumber of papers\n\nRatio (%)\n\n\n1\n\nBiochemistry\n\n62,270\n\n25.78\n\n\n2\n\nJournal of Molecular Biology\n\n29,968\n\n12.41\n\n\n3\n\nThe EMBO Journal\n\n17,296\n\n7.16\n\n\n4\n\nJournal of Theoretical Biology\n\n12,200\n\n5.05\n\n\n5\n\nBioinformatics\n\n9,847\n\n4.08\n\n\n6\n\nHuman Molecular Genetics\n\n9,347\n\n3.87\n\n\n7\n\nGenomics\n\n8,316\n\n3.44\n\n\n8\n\nBMC Genomics\n\n7,741\n\n3.20\n\n\n9\n\nBMC Bioinformatics\n\n6,780\n\n2.81\n\n\n10\n\nProtein Science: A publication of the Protein Society\n\n6,047\n\n2.50\n\n\n11\n\nJournal of Proteome Research\n\n5,575\n\n2.31\n\n\n12\n\nProteomics\n\n5,545\n\n2.30\n\n\n13\n\nJournal of Biotechnology\n\n5,204\n\n2.15\n\n\n14\n\nPLOS Genetics\n\n5,139\n\n2.13\n\n\n15\n\nPLOS Computational Biology\n\n3,852\n\n1.59\n\n\n16\n\nBMC Research Notes\n\n3,743\n\n1.55\n\n\n17\n\nMammalian Genome\n\n3,499\n\n1.45\n\n\n18\n\nGenome Biology\n\n3,411\n\n1.41\n\n\n19\n\nPLOS Biology\n\n3,280\n\n1.36\n\n\n20\n\nTrends in Biochemical Sciences\n\n3,171\n\n1.31\n\n\n21\n\nTrends in Genetics\n\n3,035\n\n1.26\n\n\n22\n\nJournal of Molecular Modeling\n\n2,852\n\n1.18\n\n\n23\n\nMolecular & cellular proteomics: MCP\n\n2,796\n\n1.16\n\n\n24\n\nTrends in Biotechnology\n\n2,353\n\n0.97\n\n\n25\n\nBulletin of Mathematical Biology\n\n2,331\n\n0.96\n\n\n26\n\nJournal of Proteomics\n\n2,158\n\n0.89\n\n\n27\n\nPhysiological Genomics\n\n1,794\n\n0.74\n\n\n28\n\nJournal of Computer-Aided Molecular Design\n\n1,706\n\n0.71\n\n\n29\n\nBMC Systems Biology\n\n1,397\n\n0.58\n\n\n30\n\nBioinformation\n\n1,297\n\n0.54\n\n\n31\n\nPharmacogenetics and Genomics\n\n1,072\n\n0.44\n\n\n32\n\nStatistical Methods in Medical Research\n\n976\n\n0.40\n\n\n33\n\nJournal of Computational Neuroscience\n\n925\n\n0.38\n\n\n34\n\nMolecular Systems Biology\n\n822\n\n0.34\n\n\n35\n\nGenome Medicine\n\n676\n\n0.28\n\n\n36\n\nTheoretical Biology and Medical Modeling\n\n498\n\n0.21\n\n\n37\n\nComparative and Functional Genomics\n\n466\n\n0.19\n\n\n38\n\nNeuroinformatics\n\n385\n\n0.16\n\n\n39\n\nCancer Informatics\n\n355\n\n0.15\n\n\n40\n\nBriefings in Functional Genomics & Proteomics\n\n290\n\n0.12\n\n\n41\n\nEvolutionary Bioinformatics\n\n249\n\n0.10\n\n\n42\n\nAlgorithms for Molecular Biology\n\n245\n\n0.10\n\n\n43\n\nJournal of Biomedical Semantics\n\n240\n\n0.10\n\n\n44\n\nBioData Mining\n\n149\n\n0.06\n\n\n45\n\nEURASIP Journal on Bioinformatics and Systems Biology\n\n140\n\n0.06\n\n\n46\n\nSource Code for Biology and Medicine\n\n131\n\n0.05\n\n\nTotal\n\n241,569\n\n100.00\n\n\n\nData preprocessing and keyphrase extraction \nWe limited the publication year back to 1996 and divided the dataset into the following four time periods to identify the trend of bioinformatics from the birth of the field to present: 1996\u20132000, 2001\u20132005, 2006\u20132010, and 2011\u20132015 (Fig. 2).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Research overflow. Data distribution. Publication year of our dataset is from 1996 to 2015. To identify topical trends of bioinformatics, we divided total 20 years into four time periods. X-axis is publication year and Y-axis is the number of papers.\n\n\n\nAs shown in Fig. 2, there is a relatively consistent increase in the number of papers. There are fewer than half as many papers published in 2015 than in 2014 because we collected our dataset in June 2015. Nevertheless, we included the 2015 data to observe the latest publication trends. \nTable 2 presents the breakdown of our dataset by period. As in Fig. 2, the fourth period is the most productive, containing 53,520 papers, or 31.46% of the total dataset. The most productive year is 2014, which accounts for 7.20% with 12,251 papers. The total number of papers for all 20 years is 170,099. This number is different from Table 1 (241,569) as a result of preprocessing; we excluded papers that did not have an abstract.\n\n\n\n\n\n\n\nTable 2. Time-based statistics for 20 years\n\n\nYear\n\nNumber of paper\n\nRatio (%)\n\nRanking\n\n\n1996\n\n5,713\n\n3.36\n\n19\n\n\n1997\n\n5,549\n\n3.26\n\n20\n\n\n1998\n\n5,853\n\n3.44\n\n18\n\n\n1999\n\n5,877\n\n3.46\n\n17\n\n\n2000\n\n5,947\n\n3.50\n\n16\n\n\nPeriod 1\n\n28,939\n\n17.01\n\n\n\n\n2001\n\n6,199\n\n3.64\n\n14\n\n\n2002\n\n6,456\n\n3.80\n\n13\n\n\n2003\n\n6,668\n\n3.92\n\n12\n\n\n2004\n\n7,564\n\n4.45\n\n11\n\n\n2005\n\n8,545\n\n5.02\n\n10\n\n\nPeriod 2\n\n35,432\n\n20.83\n\n\n\n\n2006\n\n9,845\n\n5.79\n\n9\n\n\n2007\n\n10,112\n\n5.94\n\n8\n\n\n2008\n\n10,352\n\n6.09\n\n7\n\n\n2009\n\n10,868\n\n6.39\n\n6\n\n\n2010\n\n11,031\n\n6.49\n\n5\n\n\nPeriod 3\n\n52,208\n\n30.69\n\n\n\n\n2011\n\n11,518\n\n6.77\n\n4\n\n\n2012\n\n11,986\n\n7.05\n\n2\n\n\n2013\n\n11,695\n\n6.88\n\n3\n\n\n2014\n\n12,251\n\n7.20\n\n1\n\n\n2015\n\n6,070\n\n3.57\n\n15\n\n\nPeriod 4\n\n53,520\n\n31.46\n\n\n\n\nTotal\n\n170,099\n\n100.00\n\n\n\n\n\nWe extract various metadata, such as the PMID, author, publication year, journal title, title, and abstract from XML formatted records. After XML processing, we combine the title with abstract and conduct keyphrase extraction. For keyphrase extraction, we use MAUI, which has the keyphrase model trained with MeSH terms.[28] In this dataset, there are 500 documents and several keys consisting of MeSH terms about each document, manually assigned by the indexer. MAUI is a newer version of the keyphrase extraction algorithm KEA.[29] Keyphrase extraction enables researchers to select representative phrases to make topic detection more meaningful. Therefore, we use keyphrases extracted from the title or abstract as our input for the ACT model instead of individual words.\nTable 3 shows the results of keyphrase extraction and other metadata such as the title and publication year from the PubMed record PMID 26030820.\n\n\n\n\n\n\n\nTable 3. Example of results of keyphrase extraction and other metadata from PMID of 26030820\n\n\nInformation\n\nContent\n\n\nTitle\n\nencoding cell amplitude frequency modulation\n\n\nAuthor\n\nMicali Gabriele, Aquino Gerardo, Richards David M, Endres Robert G\n\n\nYear\n\n2015\n\n\nJournal\n\nPLOS computational biology\n\n\nKeyphrases\n\nDown-Regulation - Ion Channels - Ions - L Cells (Cell Line) - Ligands - Social Control, Formal - Social Control, Informal - Up-Regulation\n\n\n\nACT model application \nThe ACT model, proposed by Tang et al.[17] as an extension of the LDA model[25], is a unified topic model for modeling various metadata simultaneously. This model starts with the assumption that the order of the topic created by the paper, author, and conference is the same. It also estimates the statistical distribution associated with all topics for the purpose of discovering latent topic distribution related with paper, author, and conference. In this paper, two metadata types are changed. First, conference is replaced with journal. Also, a bag-of-keyphrases are used instead of a bag-of-words to represent documents in a more precise manner.\nFigure 3 illustrates the ACT model, and Table 4 provides a description of the parameters used. Model estimation is conducted by setting parameters, and for estimation of the model parameter, the Gibbs sampling method is employed. Gibbs sampling takes samples from a probability distribution by using the Markov Chain Monte Carlo sampling method. Three parameters for estimating the model are as follows: 1) \u03b8 is the topic probability for a given author (author*topic matrix), 2) \u03c6 is the journal probability for a given topic (topic*journal matrix), and 3) \u03c8 is the word probability for a given topic (topic*word matrix). According to the independence assumption, joint distribution of topic, author, journal, and word stand on the basis Ad, meaning the total number of authors in paper d. In our experiments, we set the hyper-parameters \u03b1, \u03b2, and \u03b3, which are parameters of a prior with \u03b1\u2009=\u200950\/T, \u03b2\u2009=\u20090.01, and \u03b3 = 0.01, respectively. In addition, we fix the number of topics K to 20, the number of top keyphrases to 30, and the number of iterations to 1,000. With these settings, we selected 15 out of 20 topics for analysis.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. ACT Model. Author-Conference-Topic (ACT) Model is proposed by Tang et al. which is a probabilistic topic model to extract topics, authors, and conference simultaneously. \n\n\n\n\n\n\n\n\n\nTable 4. Notation and description of the ACT model\n\n\nNotation\n\nDescription\n\n\nd\n\nPaper\n\n\nx\n\nAuthor\n\n\nw\n\nWord\n\n\nj\n\nJournal\n\n\nD\n\nTotal number of papers\n\n\nA\n\nTotal number of authors\n\n\nK\n\nSelected number of topics\n\n\nNd\n\nTotal number of words in paper d\n\n\nAd\n\nTotal number of authors in paper d\n\n\nz\n\nTopic\n\n\n\u03b8\n\nAuthor-topic distribution\n\n\n\u03c6\n\nTopic-journal distribution\n\n\n\u03c8\n\nTopic-word distribution\n\n\n\u03b1,\u03b2,\u03b3\n\nHyper-parameters of Dirichlet distribution\n\n\n\nEvaluation \nTo examine the consistency of our results, we repeated each run 10 times with a topic number of 20. After that, we calculated the similarity between topics. For statistical analysis, we computed Pearson correlation coefficients between any two topics and averaged them out. Table 5 shows the average of correlation coefficients per execution. In all runs, Pearson correlation coefficients between topics were weakly, positively correlated. Also, the range of correlation was not wide (0.13 to 0.18). It implies that there was no difference in similarity between topics regardless of different runs. This result can verify consistency and reliability of our topic clusters.\n\n\n\n\n\n\n\nTable 5. Average of Pearson correlation coefficients result\n\n\nNumber of runs\n\nPearson correlation coefficients\n\n\n1\n\n0.155\n\n\n2\n\n0.140\n\n\n3\n\n0.152\n\n\n4\n\n0.177\n\n\n5\n\n0.180\n\n\n6\n\n0.146\n\n\n7\n\n0.136\n\n\n8\n\n0.160\n\n\n9\n\n0.158\n\n\n10\n\n0.178\n\n\n\nIn addition, to evaluate the topic model results, we used perplexity which is a well-known measurement in information theory for testing goodness of a model. In our case, we make a test set by collecting bioinformatics journals published in 2016. The sample size is 1,000 papers. In the training set, we divided 20 years into four periods and calculated the perplexity by setting the number of topics as 10, 20, 30, and 50 respectively. The results are presented in Table 6 and Fig. 4. As shown in Table 6 and also confirmed in Fig. 4, there is not much difference in performance in regards to the number of topics by perplexity. However, there is a clear difference among periods by perplexity. In particular, the third period has the highest perplexity value, which implies that it is the most difficult period as to predicting the topic trend in 2016 in the bioinformatics field.\n\n\n\n\n\n\n\nTable 6. Perplexity result of topic model\n\n\nNumber of topics\n\n1996\u20132000\n\n2001\u20132005\n\n2006\u20132010\n\n2011\u20132015\n\nAverage\n\n\n10\n\n2,712\n\n2,060\n\n875,088\n\n501,176\n\n345,259\n\n\n20\n\n2,978\n\n3,161\n\n726,329\n\n513,176\n\n311,411\n\n\n30\n\n2,872\n\n2,176\n\n742,307\n\n481,875\n\n307,308\n\n\n50\n\n2,480\n\n2,149\n\n635,960\n\n466,676\n\n276,816\n\n\nAverage\n\n2,760\n\n2,387\n\n744,921\n\n490,726\n\n\n\n\n\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Perplexity result. For evaluation of topic modeling results, we used perplexity. We calculated perplexity per each period with the number of topics as 10, 20, 30, and 50. X-axis is period and Y-axis means a perplexity value. \n\n\n\nTogether with this result, we analyzed the results of the ACT model.\n\nResults \nWe analyzed leading authors and journals in relation to topics over time. In the following section, we provide the detailed explanations of the trend per period.\n\nTopic analysis per period \nThe results of our time series topic analysis show that topics seem to be more distinct and subdivided closer to present. In addition, new topics have emerged in recent years, and they do not make a new cluster, which means the exclusive topics become apparent. The results also show that research fields such as molecular biology, genomics, genetics, and proteomics play a supplementary role in biology, but also become diversified into a unique field.\n\nFirst period analysis \nIn the first period (1996\u20132000), five dominant topic clusters were identified (Additional file 1: Appendix 1). Those five topics were mainly associated with proteins and peptides. Phrases such as \"molecular biology\" and \"chemical compound\" were widespread, and thermodynamics- and kinematics-related topics appeared. These topics were composed of jargon in their specific fields. The mathematical biology field was represented by topical phrases such as \"database,\" \"cluster analysis,\" \"model,\" \"theoretical,\" and \"software.\"\nTopics 0, 2, and 3 were about molecular biology, derived from biochemistry and composed of hydrogen bonding\u2013related chemical compounds such as enzymes or lipids. Topics 4, 5, 6, and 7 were related to proteins, peptides, and protein structure. Topics 9 and 14 included words such as \"probability\" and \"statistics,\" which are related to mathematical biology. Topics 13, 17, 18, and 19 covered mutagenesis, disease, and syndromes, which are all related to genetic diseases; mutagenesis consists of gene mutation, and syndromes are caused by genetic disorder. Topic 19 included the word \"genetic,\" a parent category of previously mentioned words. Topics 15 and 16 consisted of kinetics.\nProtein-related topics were dominant, and authors involved in peptide and protein structure were prevalent in the first period. Authors who were in topic 5, such as A. R. Fersht, J. M. Thornton, C. M. Dobson, L. Serrano, and M. Karplus, had a high probabilistic distribution value, which means they were leading researchers in this area. Their research interest was mainly in protein structure, and they published in the Journal of Molecular Biology. This journal appeared in almost all of the topics related to protein and dealt with structure and function of macromolecules, complexes, and protein folding.\n\nSecond period analysis \nThere were four topic clusters and one exclusive topic in the second period (Additional file 1: Appendix 2). In the second period (2001\u20132005), studies about genetics and genomics were actively conducted, and protein-related topics were diversified into subfields such as proteomics. In addition, mathematical biology and computational biology\u2013related topics were maintained in this period.\nTopics 1, 2, 5, 7, and 11 included DNA mechanisms, molecular structure, genetics, genomics, and diseases caused by DNA or a genome such as Down syndrome, DNA transposable elements, and ribonucleases. Topics 0, 3, 14, and 16 were mainly about proteomics, specifically focusing on protein structure. Topics 12, 18, and 19 addressed biotechnology, molecular modeling, and structure. Topics 8 and 9 focused on mathematical biology and computational biology. Topic 4 exclusively contained enzymology-related phrases such as \"enzyme activators\" and \"oxygen.\" Enzymology-related topics were less common compared with the first period.\nThe second period mainly focused on gene-related topics. Topic 5 had the highest probabilistic distribution among top-ranked authors such as Gregory A. Petsko, L. Aravind, Eugene V. Koonin, Mark Gerstein, and Laurence D. Hurst, who were interested in genomics and biomedical engineering. Those authors published papers in Genome Biology, which covered subject matters related to genomics and post-genomics. Similar to the first period, protein-related research was a major topic in the second period. Top-ranked authors in this topic included Ruedi Aebersold, Peter Roepstorff, Pier Giorgio Righetti, Jean-Charles Sanchez, and Peter R. Jungblut. These authors were pioneers of proteomics, and their papers were published in the Journal of Molecular Biology and Proteomics.\n\nThird period analysis \nIn the third period (2006\u20132010), topics were divided into three clusters: genomics, proteomics, and other (Additional file 1: Appendix 3). Different from the first two periods, four exclusive topics existed and seemed to be distinct from topics in the other three periods. For instance, studies about genomics or proteomics were more diversified than in the earlier periods. Exclusive topics that were not included in two large fields emerged, indicating that bioinformatics research was conducted in various fields related to bioinformatics.\nTopics 3, 7, 10, 11, 13, and 16 consisted of proteomics, protein evolution, and protein structure. Proteomics-related topics were subdivided. The representative journals in the area were Proteomics, the 'Journal of Proteome Research, and the Journal of Proteomics. Topics 5, 6, 12, 14, and 19 were gene-related topics such as gene expression, gene transcription, and genomics. Gene-related studies became prevalent in the second period. The distinct topics that appeared in the third period were topics 0, 15, 17, and 18. Topic 0, molecular biology, especially focused on hydrogen bonding. In the first and second periods, topic 15 included various topics related to theoretical biology. Topic 17 addressed hepatitis, the infection in liver cells and tissues. Different from previous periods, in the third period topics were associated with specific diseases. Topic 18 included peptide-associated phrases, and, unlike prior periods, concrete themes like specific chemical compounds and protein appeared.\nOverall, protein-related topics were most common in the third period. The third period also has more sub-divided and distinct topics than previous periods did. In this period, general topics such as proteomics appeared, as did specific topics such as protein evolution, protein analytics, and protein ubiquitin. Among these areas, the topic with the highest distribution was protein analytics, and it was sub-categorized in proteomics. Top-ranked authors in this period included Matthias Mann, Ruedi Aebersold, Richard D. Smith, Albert J. R. Heck, and Visith Thongboonkerd. They were experts in protein analytics and commonly used mass spectrometry for their analyses. They actively published in the Journal of Proteome Research and Proteomics. These two journals were top-rated in protein-related topics. The Journal of Proteome Research was computer technology\u2013oriented and focused on protein-analysis research. The journal with the highest probabilistic distribution in all topic areas was the EMBO Journal. This journal focused on molecular biology and also covered proteomics.\n\nFourth period analysis \nThe fourth period (2011\u20132015) showed three major topic clusters and two exclusive topics (Additional file 1: Appendix 4). Similar to the third period, the topics related with genomics and proteomics were further divided into subfields and represented concrete topical characteristics. Compared with the third period\u2019s results, theoretical biology\u2013related topics formed one cluster. The compositions of the cluster were one big topic (systems biology) and four sub-divided topics.\nTopics 1 and 16 were theoretical biology\u2013related, and topics 6 and 10 were about systems biology. They could be clustered as a broader category of system biology. The representative journals in this cluster were PLOS Computational Biology, Journal of Theoretical Biology, and Journal of Computational Neuroscience, which were focused on systems biology. Topics 0, 11, 12, 18, and 19 were about genetics and genomics. Topics 4, 9, 13, and 17 represented proteomics. Topics 8 and 15 were exckusive, each of which was related to molecular biology and cell biology. Topic 8 included phrases like \"hydrogen bonding\" and \"GTP-binding proteins,\" and topic 15 contained phrases like \"enteroendocrine cells\" and \"COS cells.\" The top journals in these areas were Biochemistry, the Journal of Molecular Biology, and the Journal of Molecular Modeling.\nIn the fourth period, the major topics were systems biology, genomics, and proteomics. Topics that were not in the main stream of bioinformatics were found in this period, and topics about theoretical biology and systems biology become a distinct cluster. This means that these areas were growing in the bioinformatics area. The representative researchers in this area were Martin A. Nowak, Yoh Iwasa, Mike Steel, Ulf Dieckmann, and Liam Paninski. They were mostly involved in mathematics and theoretical biology. The journal which had the highest probabilistic distribution was the Journal of Theoretical Biology. This journal focused on research that combines biology and topics such as statistical analysis, mathematical definition, comparative research, experiments, and computer simulation. The second raked journal was the Journal of Bioinformatics, which mainly accepted research about genome bioinformatics and computational biology.\n\nDiscussion \nIn this section, we analyze the results from three different perspectives: topical keyphrase, journal, and author. In addition, to further identify which researchers and journals focus on which topic over time, the results of the ACT model (top-ranked keyphrases, authors, and journals) are examined in an integrated perspective.\n\nTime series analysis \nOne interesting observation is that keyphrases related with genes or genetic processes such as \"gene expression,\" \"down-regulation,\" and \"up-regulation\" were not ranked high in the first period. However, they emerged as top keyphrases in later periods. In particular, \"proteome,\" \"reproducibility of results,\" \"proteomics,\" and \"genotype\" did not appear in the first period but emerged gradually after that. In addition, the term \"algorithms\" appeared most frequently in the fourth period. In author perspective, across the four periods, the number of unique authors was 1,396. Top ranked author Richard Robinson appeared in five topics. Seven authors, including Liza Gross, appeared four times, 45 authors appeared three times, 137 authors appeared twice, and the remaining 1,184 authors were shown only in one topic. There was no author who appeared in all four periods. Thirty-nine authors appeared across three periods, 125 authors appeared in two periods, and 1,210 authors appeared only in a period. In journal centered view, only 21 out of 46 journals appeared in the first period. In the second and third period, 34 and 46 journals were presented respectively. Forty-five journals appeared in the last period; one the journals, Briefings in Functional Genomics & Proteomics, did not appear in the last period.\nThese results imply that the bioinformatics field is diversified and new topical disciplines have recently emerged. For instance, proteomics-related topics started to appear in the second period, became segmented into detailed research fields and later evolved in the third and fourth periods. In addition, while conceptual biology\u2013related topics existed in the first period, they became clearly progressed in the fourth period. Conversely, the topics about kinetics appeared in the first period but then faded out.\n\nIntegrated view of graph pattern analysis \nFor further integrated analysis, we examined top journals with their topical probability in all four periods. We also checked the authors and topical keyphrases which were topically matched with the journals. We identified that there were four different patterns in a journal\u2019s topical distribution such as rising, falling, concave, and convex patterns. In Fig. 5, we only presented graphs which drastically changed in terms of the probability value of topics between periods. Additionally in each graph, we presented the top five ranked authors and keyphrases which had a high probability value across four periods.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Journal focused topic distribution with related authors and keyphrases. For integrated pattern analysis, we examined eight representative journals with top authors and keyphrases. Patterns were classified as four outstanding ones such as rising (a-b), falling (c-f), convex \ufeff(g) and concave (h).\n\n\n\nWe explained four outstanding cases in each pattern and made a list of journals which are shown in all four periods (Table 7). First for the rising pattern, the journal BMC Bioinformatics had 0.86060 gaps between maximum and minimum probability. It was the highest gap by probability in all rising patterns. The average impact factor of this journal provided by journal citation report (JCR) was 3.0806 in 2015. In this context, BMC Bioinformatics could be regarded as the most promising journal in the bioinformatics field. The journal has grown steadily through 20 years. The authors belonging to the similar topical scope of BMC Bioinformatics were presented in graph (a). The top ranked authors shared common characteristics: the authors were majored in computer science or statistics and then applied the technique to biomedical or biology later on, and their common research interests were bioinformatics or biostatistics. As shown in graph (a), the topical keyphrases related with the journal were not focused only on biology research fields. The word \"algorithms\" represents the informatics field, while \"genome,\" \"genomics,\" and \"gtp-binding proteins\" relate to the genomics fields, and the word \"protein s\" represents protein- or gene-related scopes. The scope of the journal is in computational and statistical methods for the modeling and analysis of all kinds of biological data, as well as other areas of computational biology. The results indicated that the publication trend of BMC Bioinformatics changed to genetics or genomics converged with informatics. \nIn the case of the falling pattern of the journal Biochemistry, as shown in graph (c), the journal had 4.29181 gap between the maximum and minimum value. The average impact factor of this journal in four periods was 3.75322. The journal had a somewhat high probability value, but the impact factor in each period decreased gradually (e.g., first period: 4.4785 to fourth period: 3.1768). This decreasing pattern implied that in bioinformatics field, the journal dealt mainly with biochemistry, biophysical chemistry, and molecular biology, but it had not converged with informatics. That means the biochemistry field did not have much interaction with informatics fields. \nThe top ranked authors shown in graph (c) were celebrated scholars in the biochemistry field. Commonly, they all researched only the biochemistry field, and we could not find that they had a connection with the informatics fields. In the same context, top ranked keyphrases of the journal were mostly related to biochemistry research fields. The informatics-related keyphrase did not appear in the top word list.\n\n\n\n\n\n\n\nTable 7. The list of journals showed in all periods\n\n\nJournal name\n\nSum of probability\n\nAverage of probability\n\n\nBiochemistry\n\n13.13233\n\n3.28308\n\n\nBioinformatics (Oxford, England)\n\n2.49214\n\n0.62304\n\n\nBMC Bioinformatics\n\n1.64624\n\n0.41156\n\n\nBMC Genomics\n\n1.92022\n\n0.48005\n\n\nBulletin of Mathematical Biology\n\n0.68576\n\n0.17144\n\n\nGenome Biology\n\n1.28690\n\n0.32173\n\n\nGenomics\n\n2.40453\n\n0.60113\n\n\nHuman Molecular Genetics\n\n2.98031\n\n0.74508\n\n\nJournal of Biotechnology\n\n1.32392\n\n0.33098\n\n\nJournal of Computational Neuroscience\n\n0.42877\n\n0.10719\n\n\nJournal of Computer-Aided Molecular Design\n\n0.60322\n\n0.15081\n\n\nJournal of Molecular Biology\n\n5.82534\n\n1.45633\n\n\nJournal of Theoretical Biology\n\n3.00820\n\n0.75205\n\n\nMammalian Genome\n\n1.06714\n\n0.26678\n\n\nPhysiological Genomics\n\n0.61599\n\n0.15400\n\n\nProtein Science\n\n2.16372\n\n0.54093\n\n\nStatistical Methods in Medical Research\n\n0.25062\n\n0.06266\n\n\nThe EMBO Journal\n\n3.35207\n\n0.83802\n\n\nTrends in Biochemical Sciences\n\n0.50571\n\n0.12643\n\n\nTrends in Biotechnology\n\n0.50871\n\n0.12718\n\n\nTrends in Genetics\n\n0.73352\n\n0.18338\n\n\n\nNot only the rising and falling patterns, but the convex and concave patterns of journals exist. In graph (g), BMC Genomics was the journal showing the convex pattern. BMC Genomics dealt with genome-scale analysis, functional genomics, and proteomics. The average impact factor from 2000 to 2015 was 4.0464. The topical probability of the journal rose from the first period (0.00125) to the third period (1.19977), but since then the probability dropped to 0.51576. The gap between maximum and minimum value was 1.19852. The journal changed the status to open-access in 2000, and maybe it caused a slight rise from first period to second period. For the drastic increase from second period to third period, the number of total citations increased approximately two times according to the JCR. The increasing and decreasing number in total citations may cause the wave in graph (g). The top five ranked authors\u2019 research interests had various characteristics. N. A. Jenkins and N.G. Copeland were a couple, with Jenkins interested in molecular and cellular biology, while the husband Copeland was in the biochemistry field. They collaborated a lot and both were celebrated researchers in the field. The other researchers also had different research areas, including treatment or preventive therapy (Gregory A. Petsko), genomic medicine (Y. Nakamura), and genome analysis (L. Aravind). Through all four periods, the journal mainly published articles which dealt with the genomic related keyphrases. The authors and keyphrases located in this journal were not quite related to the informatics fields. \nLast, the concave pattern was shown in graph (h), and Human Molecular Genetics was the representative journal. The journal had steadily decreasing impact factor flow in 20 years (e.g., first period: 9.05475 to fourth period: 6.8766). The gap between maximum and minimum probability was 0.81973. In graph (h), the point of drastic decrease can be readily observed. Top ranked authors related with the journal focused on genetics. N. G. Copeland and N. A. Jenkins appeared again. The rest of the authors did not overlap in research areas. The journal was interested in broad genetics-related topical keyphrases but not related with informatics fields.\nThrough the pattern analysis integrated with journal, author, and keyphrase, we identified that the field of bioinformatics was a converging area, and certain journals clearly showed rising and falling patterns. Different from period analysis, the integrated view of analysis showed journals\u2019 topical trends over time along with top journals and researchers.\n\nConclusions \nBioinformatics mainly tackles biological problems at the molecular level using applied mathematics, information science, statistics, computer science, chemistry, and biochemistry. This characteristic of bioinformatics has driven the field to become interdisciplinary, combining approaches from various fields to make use of a large amount of data.\nIn this study, we investigated the bioinformatics field using the ACT model to conduct comprehensive topic analyses of keyphrases, authors, and journals. To this end, we collected 46 journals belonging to the bioinformatics field by searching journal name in PubMed, yielding 170,099 papers. To analyze topic evolution over time, we divided the collected datasets into four, five-year periods: 1996\u20132000, 2001\u20132005, 2006\u20132010, and 2011\u20132015. In a time series topic analysis, we examined topic clusters within periods. In more recent periods, distinct characteristics of the field emerged and more new topics were presented independently. In addition, we analyzed trends in keyphrases, authors, and journals. Our keyphrase analysis similarly indicated the emergence of greater interdisciplinary research over time. In our author analysis, we observed the pattern of authors who appear in top rank. In our journal analysis, we analyzed the common topic area of top journals and identified major focus areas of those top journals, including computational biology, theoretical biology, and mathematical biology. Also, we examined topic distribution over journals with top ranked authors and keyphrases. In this analysis, journals were identified as four patterns over time such as rising, falling, convex, and concave patterns.\nThe results of these analyses imply that the bioinformatics field is highly interdisciplinary, consisting of active convergence studies. In addition, we observed that the characteristics of the bioinformatics field become more distinct and more specific, and the supporting role of peripheral fields of bioinformatics, such as conceptual, mathematical, and systems biology, gradually increases over time, though the core fields of proteomics, genomics, and genetics are still the major topics. This is consistently confirmed by analysis of topic distribution of journals over time as well as integrated analysis of top ranked keyphrases, authors, and journals.\nIn the future, we plan to apply the same approach to other domains, such as information science. We also plan to use other metadata such as MeSH terms and bio-entities to compare with keyphrases. In addition, we plan to explore how to infer authors\u2019 interests by time series analysis and identify representative authors and the journals that are the best suited for a paper on a particular subject.\n\nDeclarations \nAcknowledgements \nNot applicable.\n\nFunding \nThis work was supported by the Bio-Synergy Research Project (NRF-2013M3A9C4078138) of the Ministry of Science, ICT and Future Planning through the National Research Foundation. This work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. NRF-2014R1A2A2A01004454). The publication charges for this article was funded by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP).\n\nAvailability of data and materials \nNot applicable.\n\nAuthors' contributions \nGEH conceived and designed the study. GEH and KYK conducted the analysis and drafted the manuscript. MS designed and coordinated the study. All authors contributed to the writing of the manuscript. All authors read and approved the final manuscript.\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nConsent for publication \nNot applicable.\n\nEthics approval and consent to participate \nNot applicable.\n\nAbout this supplement \nThis article has been published as part of BMC Bioinformatics Volume 18 Supplement 7, 2017: Proceedings of the Tenth International Workshop on Data and Text Mining in Biomedical Informatics. The full contents of the supplement are available online at https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/supplements\/volume-18-supplement-7.\n\nPublisher\u2019s note \nSpringer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.\n\nAdditional files \n Additional File 1: Appendix 1. ACT model results in first period. Appendix 2. ACT model results in second period. Appendix 3. ACT model results in third period. Appendix 4. ACT model results in fourth period. (DOCX 48 kb)\nReferences \n\n\n\u2191 Dong, D.; Chen, M.-L. (2015). \"Publication trends and co-citation mapping of translation studies between 2000 and 2015\". Scientometrics 105 (2): 1111\u20131128. doi:10.1007\/s11192-015-1769-1.   \n\n\u2191 2.0 2.1 Chen, H.; Wan, Y.; Jiang, S.; Cheng, Y. (2014). \"Alzheimer\u2019s disease research in the future: bibliometric analysis of cholinesterase inhibitors from 1993 to 2012\". Scientometrics 98 (3): 1865\u20131877. doi:10.1007\/s11192-013-1132-3.   \n\n\u2191 3.0 3.1 Soteriades, E.S.; Falagas, M.E. (2006). \"Alzheimer\u2019s disease research in the future: Bibliometric analysis of cholinesterase inhibitors from 1993 to 2012\". BMC Public Health 6: 301. doi:10.1186\/1471-2458-6-301. PMC PMC1766930. PMID 17173665. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1766930 .   \n\n\u2191 4.0 4.1 Ugolini, D.; Puntoni, R.; Perera, F.P. et al. (2007). \"A bibliometric analysis of scientific production in cancer molecular epidemiology\". Carcinogenesis 28: 8. doi:10.1093\/carcin\/bgm129. PMID 17548902.   \n\n\u2191 Wang, L.; Chen, X.; Bao, A. et al. (2015). \"A bibliometric analysis of research on Central Asia during 1990\u20132014\". Scientometrics 105 (2): 1223\u20131237. doi:10.1007\/s11192-015-1727-y.   \n\n\u2191 6.0 6.1 Bornmann, L.; Mutz, R. (2015). \"Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references\". Journal of the Association for Information Science and Technology 66 (11): 2215\u20132222. doi:10.1002\/asi.23329.   \n\n\u2191 7.0 7.1 Geaney, F.; Scutaru, C.; Kelly, C. et al. (2015). \"Type 2 Diabetes Research Yield, 1951-2012: Bibliometrics Analysis and Density-Equalizing Mapping\". PLoS One 10: 7. doi:10.1371\/journal.pone.0133009. PMC PMC4514795. PMID 26208117. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4514795 .   \n\n\u2191 8.0 8.1 Mac\u00edas-Chapula, C.A.; Mijangos-Nolasco, A. (2002). \"Bibliometric analysis of AIDS literature in Central Africa\". Scientometrics 54 (2): 309\u2013317. doi:10.1023\/A:1016074230843.   \n\n\u2191 9.0 9.1 Seglen, P.O.; Aksnes, D.W. (2000). \"Scientific Productivity and Group Size: A Bibliometric Analysis of Norwegian Microbiological Research\". Scientometrics 49 (1): 125\u2013143. doi:10.1023\/A:1005665309719.   \n\n\u2191 10.0 10.1 Jeong, D.-H.; Song, M. (2014). \"Time gap analysis by the topic model-based temporal technique\". Journal of Infometrics 8 (3): 776\u2013790. doi:10.1016\/j.joi.2014.07.005.   \n\n\u2191 11.0 11.1 11.2 Song, M.; Kim, S.Y. (2013). \"Detecting the knowledge structure of bioinformatics by mining full-text collections\". Scientometrics 96 (1): 183\u2013201. doi:10.1007\/s11192-012-0900-9.   \n\n\u2191 12.0 12.1 Song, M.; Kim, S.Y.; Zhang, G. et al. (2014). \"Productivity and influence in bioinformatics: A bibliometric analysis using PubMed central\". Journal of the Association for Information Science and Technology 65 (2): 352\u2013371. doi:10.1002\/asi.22970.   \n\n\u2191 13.0 13.1 Yan, E. (2015). \"Research dynamics, impact, and dissemination: A topic-level analysis\". Journal of the Association for Information Science and Technology 66 (1): 2357\u20132372. doi:10.1002\/asi.23324.   \n\n\u2191 14.0 14.1 Steyvers, M.; Smyth, P.; Rosen-Zvi, M.; Griffiths, T. (2004). \"Probabilistic author-topic models for information discovery\". Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2004: 306\u2013315. doi:10.1145\/1014052.1014087.   \n\n\u2191 15.0 15.1 Li, D.; Okamoto, J.; Liu, H.; Leischow, S. (2015). \"A bibliometric analysis on tobacco regulation investigators\". BioData Mining 8: 11. doi:10.1186\/s13040-015-0043-7. PMC PMC4432889. PMID 25984237. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4432889 .   \n\n\u2191 16.0 16.1 Rosen-Zvi, M.; Griffiths, T., Steyvers, M.; Smyth, P. (2004). \"The author-topic model for authors and documents\". Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence 2004: 487\u2013494. ISBN 0974903906.   \n\n\u2191 17.0 17.1 17.2 Tang, J.; Zhang, J.; Yao, L. et al. (2008). \"ArnetMiner: Extraction and mining of academic social networks\". Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2008: 990-998. doi:10.1145\/1401890.1402008.   \n\n\u2191 Ramos, J.M.; Padilla, S.; Masi\u00e1, M.; Guti\u00e9rrez, F. (2008). \"A bibliometric analysis of tuberculosis research indexed in PubMed, 1997-2006\". International Journal of Tuberculosis and Lung Disease 2008: 1461\u20138. PMID 19017458.   \n\n\u2191 Claude, R.; Charles-Daniel, A.; Jean, A.; Jean-Francois, G. (2004). \"Bibliometric overview of the utilization of artificial neural networks in medicine and biology\". Scientometrics 59 (1): 117\u2013130. doi:10.1023\/B:SCIE.0000013302.59845.34.   \n\n\u2191 Patra, S.K.; Mishra, S. (2006). \"Bibliometric study of bioinformatics literature\". Scientometrics 67 (3): 477\u2013489. doi:10.1556\/Scient.67.2006.3.9.   \n\n\u2191 Song, M.; Heo, G.E.; Kim, S.Y. (2014). \"Analyzing topic evolution in bioinformatics: Investigation of dynamics of the field with conference data in DBLP\". Scientometrics 101 (1): 397\u2013428. doi:10.1007\/s11192-014-1246-2.   \n\n\u2191 Lee, D.; Kim, W.C.; Charidimou, A.; Song, M. (2015). \"A Bird's-Eye View of Alzheimer's Disease Research: Reflecting Different Perspectives of Indexers, Authors, or Citers in Mapping the Field\". Journal of Alzheimer's Disease 45 (4): 1207-22. doi:10.3233\/JAD-142688. PMID 25697702.   \n\n\u2191 Ding, Y.; Song, M.; Han, J. et al. (2013). \"Entitymetrics: Measuring the impact of entities\". PLoS One 8 (8): e71416. doi:10.1371\/journal.pone.0071416. PMC PMC3756961. PMID 24009660. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3756961 .   \n\n\u2191 Hofmann, T. (1999). \"Probabilistic latent semantic indexing\". Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1999: 50\u201357. doi:10.1145\/312624.312649.   \n\n\u2191 25.0 25.1 Blei, D.M.; Ng, A.Y.; Jordan, M.I. (2003). \"Latent dirichlet allocation\". Journal of Machine Learning Research 3 (1): 993\u20131022. http:\/\/jmlr.csail.mit.edu\/papers\/v3\/blei03a.html .   \n\n\u2191 Tang, J.; Zhang, J.; Jin, R. et al. (2011). \"Topic level expertise search over heterogeneous networks\". Machine Learning 82 (2): 211\u2013237. doi:10.1007\/s10994-010-5212-9.   \n\n\u2191 Kim, H.J.; An, J.; Jeong, Y.K.; Song, M. (2016). \"Exploring the Leading Authors and Journals in Major Topics by Citation Sentences and Topic Modeling\". Proceedings of the Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL) co-located with the Joint Conference on Digital Libraries 2016 2016: 42\u201350. http:\/\/dblp.uni-trier.de\/db\/conf\/jcdl\/birndl2016.html .   \n\n\u2191 Medelyan, O. (2009). \"Human-competitive automatic topic indexing (Thesis)\". The University of Waikato. http:\/\/hdl.handle.net\/10289\/3513 .   \n\n\u2191 Witten, I.H.; Paynter, G.W.; Frank, E. et al. (1999). \"KEA: Practical automatic keyphrase extraction\". Proceedings of the Fourth ACM Conference on Digital Libraries 1999: 254\u2013255. doi:10.1145\/313238.313437.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Grammar and word used were updated to make the text easier to read.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\">https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 13 June 2017, at 19:55.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 8,053 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","55740ce4aa95ae034b56b161256958e3_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Analyzing the field of bioinformatics with the multi-faceted topic modeling technique<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">Bioinformatics<\/a> is an interdisciplinary field at the intersection of molecular biology and computing technology. To characterize the field as a convergent domain, researchers have used bibliometrics, augmented with text-mining techniques for content analysis. In previous studies, Latent Dirichlet Allocation (LDA) was the most representative topic modeling technique for identifying topic structure of subject areas. However, as opposed to revealing the topic structure in relation to metadata such as authors, publication date, and journals, LDA only displays the simple topic structure.\n<\/p><p><b>Methods<\/b>: In this paper, we adopt the Author-Conference-Topic (ACT) model of Tang <i>et al.<\/i> to study the field of bioinformatics from the perspective of keyphrases, authors, and journals. The ACT model is capable of incorporating the paper, author, and conference into the topic distribution simultaneously. To obtain more meaningful results, we used journals and keyphrases instead of conferences and the bag-of-words. For analysis, we used PubMed to collect forty-six bioinformatics journals from the MEDLINE database. We conducted time series topic analysis over four periods from 1996 to 2015 to further examine the interdisciplinary nature of bioinformatics.\n<\/p><p><b>Results<\/b>: We analyzed the ACT Model results in each period. Additionally, for further integrated analysis, we conducted a time series analysis among the top-ranked keyphrases, journals, and authors according to their frequency. We also examined the patterns in the top journals by simultaneously identifying the topical probability in each period, as well as the top authors and keyphrases. The results indicate that in recent years diversified topics have become more prevalent, and convergent topics have become more clearly represented.\n<\/p><p><b>Conclusion<\/b>: The results of our analysis imply that over time the field of bioinformatics becomes more interdisciplinary where there is a steady increase in peripheral fields such as conceptual, mathematical, and system biology. These results are confirmed by integrated analysis of topic distribution as well as top ranked keyphrases, authors, and journals.\n<\/p><p><b>Keywords<\/b>: bioinformatics, text mining, topic modeling, ACT model, keyphrase extraction\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Over the years, academic subject areas have converged to form a variety of new, interdisciplinary fields. Bioinformatics is one example. Research domains from molecular biology to machine learning are used in conjunction to better understand complex biological systems such as cells, tissues, and the human body. Due to the complexity and broadness of the field, bibliometric analysis is often adopted to assess the current knowledge structure of a subject area, specify the current research themes, and identify the core literature of that area.<sup id=\"rdp-ebb-cite_ref-DongPublic15_1-0\" class=\"reference\"><a href=\"#cite_note-DongPublic15-1\" rel=\"external_link\">[1]<\/a><\/sup>\n<\/p><p>Bibliometrics identifies research trends using quantitative measures such as a researcher\u2019s number of publications and citations, journal impact factors, and other indices that can measure impact or productivity of author or journal.<sup id=\"rdp-ebb-cite_ref-ChenAlzh14_2-0\" class=\"reference\"><a href=\"#cite_note-ChenAlzh14-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SoteriadesABiblio06_3-0\" class=\"reference\"><a href=\"#cite_note-SoteriadesABiblio06-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-UgoliniABiblio07_4-0\" class=\"reference\"><a href=\"#cite_note-UgoliniABiblio07-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WangABiblio15_5-0\" class=\"reference\"><a href=\"#cite_note-WangABiblio15-5\" rel=\"external_link\">[5]<\/a><\/sup> In addition, other factors such as the affiliation of authors, collaborations, and citation data are often incorporated into bibliometric analysis.<sup id=\"rdp-ebb-cite_ref-BornmannGrowth15_6-0\" class=\"reference\"><a href=\"#cite_note-BornmannGrowth15-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GeaneyType2Dia15_7-0\" class=\"reference\"><a href=\"#cite_note-GeaneyType2Dia15-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Mac.C3.ADas-ChapulaBiblio02_8-0\" class=\"reference\"><a href=\"#cite_note-Mac.C3.ADas-ChapulaBiblio02-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SeglenScientific00_9-0\" class=\"reference\"><a href=\"#cite_note-SeglenScientific00-9\" rel=\"external_link\">[9]<\/a><\/sup>\n<\/p><p>Previous studies mainly rely on quantitative measures and suffer from the lack of content analysis. To incorporate content analysis into bibliometrics, text-mining techniques are applied. Topic-modeling techniques are mostly adopted to identify the topics of a subject area while analyzing that area more abundantly.<sup id=\"rdp-ebb-cite_ref-JeongTimeGap14_10-0\" class=\"reference\"><a href=\"#cite_note-JeongTimeGap14-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SongDetect13_11-0\" class=\"reference\"><a href=\"#cite_note-SongDetect13-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SongProduct13_12-0\" class=\"reference\"><a href=\"#cite_note-SongProduct13-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-YanResearch15_13-0\" class=\"reference\"><a href=\"#cite_note-YanResearch15-13\" rel=\"external_link\">[13]<\/a><\/sup> These techniques allow for enriched content analysis. As an extension of Latent Dirichlet Allocation (LDA), which is the best received topic-modeling technique, Steyvers <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SteyversProbab04_14-0\" class=\"reference\"><a href=\"#cite_note-SteyversProbab04-14\" rel=\"external_link\">[14]<\/a><\/sup> proposed the author-topic modeling technique that analyzes authors and topics simultaneously. They identify the authors\u2019 impact or productivity of researchers in a given subject area.<sup id=\"rdp-ebb-cite_ref-LiABiblio15_15-0\" class=\"reference\"><a href=\"#cite_note-LiABiblio15-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Rosen-ZviTheAuthor04_16-0\" class=\"reference\"><a href=\"#cite_note-Rosen-ZviTheAuthor04-16\" rel=\"external_link\">[16]<\/a><\/sup> By adding multiple conditions to LDA, Tang <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TangArnetMiner08_17-0\" class=\"reference\"><a href=\"#cite_note-TangArnetMiner08-17\" rel=\"external_link\">[17]<\/a><\/sup> suggested a new methodology, called the Author-Conference-Topic (ACT) model that analyzes the author, conference, and topic in one model to understand the subject area in an integrated manner.\n<\/p><p>In this paper, we apply the ACT model to examine the interdisciplinary nature of bioinformatics. Unlike studies that use extended versions of LDA for topic analysis, the ACT model enables us to analyze topic, author, and journal at one time, providing an integrated view for understanding bioinformatics. The research questions that we are to investigate in this paper are: 1) What are the topical trends of bioinformatics over time? 2) Who are the key contributors in major topics of bioinformatics?, and 3) Which journal is leading which topic?\n<\/p><p>To address these questions, we collect PubMed articles in XML format and extract metadata and content such as the PMID, author, year, journal, title, and abstract. From the title and abstract, we extract keyphrases, which provide more meaningful interpretations than single words, as an input of the ACT model. We also divide the collected datasets into four time periods to examine the topic changes over time. The results of ACT model\u2013based analysis show that various topics begin to appear and mixed subject topics become more apparent over time.\n<\/p><p>The rest of the paper is organized as follows. In the \"Related work\" section, we discuss work related to bibliometric analysis and topic modeling. We then describe the proposed method in the \"Methods\" section. We analyze and discuss the results of leading topics, authors, and journals in the \"Results\" and \"Discussion\" sections. Finally, we conclude the paper and suggest future lines of inquiry in the \"Conclusion\" section.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Related_work\">Related work<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Bibliometric_analysis\">Bibliometric analysis<\/span><\/h3>\n<p>Bibliometric analysis identifies the research trends in a given subject area and core journals or documents, and it helps with contrasting analysis. Many bibliometric studies use the number of published articles or journal impact factors to measure research productivity or to identify core journals in a specific field. Soteriades and Falalgas<sup id=\"rdp-ebb-cite_ref-SoteriadesABiblio06_3-1\" class=\"reference\"><a href=\"#cite_note-SoteriadesABiblio06-3\" rel=\"external_link\">[3]<\/a><\/sup> applied quantitative and qualitative measurements to analyze the fields of preventive medicine, occupational and environmental medicine, epidemiology, and public health using the number of articles and impact factor. Ugolini <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-UgoliniABiblio07_4-1\" class=\"reference\"><a href=\"#cite_note-UgoliniABiblio07-4\" rel=\"external_link\">[4]<\/a><\/sup> measured research productivity and evaluated the publication trends in the field of cancer molecular epidemiology. To quantify productivity, they used the number of articles and the average and sum of impact factors. To evaluate publication trends, they collected and divided the keywords from MeSH terms about the publication into six groups. Ramos <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-RamosABiblio08_18-0\" class=\"reference\"><a href=\"#cite_note-RamosABiblio08-18\" rel=\"external_link\">[18]<\/a><\/sup> measured the national research activity of the tuberculosis field, using impact factor and the first author\u2019s address. Claude <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ClaudeBiblio04_19-0\" class=\"reference\"><a href=\"#cite_note-ClaudeBiblio04-19\" rel=\"external_link\">[19]<\/a><\/sup> examined research productivity by using distribution of publications related to medicine and ANN, the subfield of biology. They used the number of publications, impact factor, and journal category compared with national gross domestic product (GDP). In the bioinformatics field, Patra and Mishra<sup id=\"rdp-ebb-cite_ref-PatraBiblio06_20-0\" class=\"reference\"><a href=\"#cite_note-PatraBiblio06-20\" rel=\"external_link\">[20]<\/a><\/sup> used the number of articles, publication of each journal, publication type, and the impact factor of journals to understand the growth of bioinformatics. They also found the core journals in the bioinformatics fields. Using author affiliation, they applied Lotka\u2019s law to assess the distribution of each author\u2019s productivity. Chen <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ChenAlzh14_2-1\" class=\"reference\"><a href=\"#cite_note-ChenAlzh14-2\" rel=\"external_link\">[2]<\/a><\/sup> identified research trends using statistical methods based on the type of publication, language, and distribution of nation or institution. They measured h-index, adding statistical materials with the number of citations. Through this, they analyzed the research productivity by topic, institution, and journal. In addition, they conducted a keyword analysis to comprehend the research trend in a macroscopic view.\n<\/p><p>Mainstream bibliometrics research focuses on identifying the knowledge structure of a certain field with quantitative measures. In addition, some studies use author information or the collaboration pattern among authors to understand the certain field. Seglen and Aksnes<sup id=\"rdp-ebb-cite_ref-SeglenScientific00_9-1\" class=\"reference\"><a href=\"#cite_note-SeglenScientific00-9\" rel=\"external_link\">[9]<\/a><\/sup> used the size and the productivity of research groups in the microbiology field in Norway as a measurement for bibliometric analysis. Geaney <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-GeaneyType2Dia15_7-1\" class=\"reference\"><a href=\"#cite_note-GeaneyType2Dia15-7\" rel=\"external_link\">[7]<\/a><\/sup> performed bibliometric analysis and density-equalizing mapping on scientific publications related to type 2 diabetes mellitus. They collected citation data and used various citation-oriented measures such as the number of citations, the average number of citations per journal, the total number of publications, impact factor, and Eigenfactor score. To conduct content analysis and study the collaboration pattern between authors and the core sub-field of AIDS, Mac\u00edas-Chapula and Mijangos-Nolasco<sup id=\"rdp-ebb-cite_ref-Mac.C3.ADas-ChapulaBiblio02_8-1\" class=\"reference\"><a href=\"#cite_note-Mac.C3.ADas-ChapulaBiblio02-8\" rel=\"external_link\">[8]<\/a><\/sup> analyzed MeSH thesaurus using check tags, main headings, and subheadings of each MeSH term hierarchy. In addition, to measure the national research productivity, they used the authors\u2019 address information. Bornmann and Mutz<sup id=\"rdp-ebb-cite_ref-BornmannGrowth15_6-1\" class=\"reference\"><a href=\"#cite_note-BornmannGrowth15-6\" rel=\"external_link\">[6]<\/a><\/sup> recently identified the development of modern science by bibliometric analysis. They divide the data into three time periods to analyze the changes of fields over time.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Text_mining_applied_to_bibliometrics\">Text mining applied to bibliometrics<\/span><\/h3>\n<p>Recently, there have been many attempts to apply text-mining techniques to bibliometric analysis to identify the knowledge structure of the field or measure its influence on other researchers and their fields and productivity. Song and Kim<sup id=\"rdp-ebb-cite_ref-SongDetect13_11-1\" class=\"reference\"><a href=\"#cite_note-SongDetect13-11\" rel=\"external_link\">[11]<\/a><\/sup> collected full-text articles from PubMed Central and computed their citation relation. They infer the knowledge structure and understand the trend of the bioinformatics field. In a similar vein, Song <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SongProduct13_12-1\" class=\"reference\"><a href=\"#cite_note-SongProduct13-12\" rel=\"external_link\">[12]<\/a><\/sup> measured the influence and productivity of bioinformatics by mining full-text articles retrieved from PubMed Central. To calculate the field\u2019s productivity, they identified the most productive author, nation, institution, and topic word; to calculate its influence, they identified the most-cited paper, author, and rising researcher. Song <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SongAnal14_21-0\" class=\"reference\"><a href=\"#cite_note-SongAnal14-21\" rel=\"external_link\">[21]<\/a><\/sup> analyzed topic evolution in the bioinformatics field using DBLP data in the field of computer science. To identify topic trends over time, they divided a dozen years (2000\u20132011) into four periods and applied Markov Random Field-based topic clustering. For automatic clustering labeling, they calculated topic similarity based on Within-Period Cluster Similarity (WPCS) and Between-Period Cluster Similarity (BPCS). Their approach created topic graphs that show interaction among topics over some period of time. Lee <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-LeeABird15_22-0\" class=\"reference\"><a href=\"#cite_note-LeeABird15-22\" rel=\"external_link\">[22]<\/a><\/sup> mapped the Alzheimer\u2019s disease field into three different perspectives: indexer, author, and citer. They applied entity-metrics<sup id=\"rdp-ebb-cite_ref-DingEntity13_23-0\" class=\"reference\"><a href=\"#cite_note-DingEntity13-23\" rel=\"external_link\">[23]<\/a><\/sup> the extended notion of bibliometrics, to analyze the field by constructing four kinds of networks that convey these three perspectives.\n<\/p><p>These studies identify the knowledge structure of a certain field by constructing bibliometric networks or databases with text-mining techniques. The most prevalent approach is to apply topic modeling to content analysis as a part of bibliometrics. Starting from the probabilistic Latent Semantic Indexing (pLSI)<sup id=\"rdp-ebb-cite_ref-HofmannProb99_24-0\" class=\"reference\"><a href=\"#cite_note-HofmannProb99-24\" rel=\"external_link\">[24]<\/a><\/sup> model, Latent Dirichlet Allocation (LDA)<sup id=\"rdp-ebb-cite_ref-BleiLatent03_25-0\" class=\"reference\"><a href=\"#cite_note-BleiLatent03-25\" rel=\"external_link\">[25]<\/a><\/sup> is the most accepted topic modeling technique for bibliometrics. While each document consists of a set of topics in pLSI, using the LDA model results in a more precise manipulation being added to organize the topics. Yan<sup id=\"rdp-ebb-cite_ref-YanResearch15_13-1\" class=\"reference\"><a href=\"#cite_note-YanResearch15-13\" rel=\"external_link\">[13]<\/a><\/sup> used the LDA model to measure the influence and popularity of library and information science. He also identified the most-cited area and the patterns in this field. Jeong and Song\u2019s<sup id=\"rdp-ebb-cite_ref-JeongTimeGap14_10-1\" class=\"reference\"><a href=\"#cite_note-JeongTimeGap14-10\" rel=\"external_link\">[10]<\/a><\/sup> research measured the time gap among three different resources \u2014 web, patent, and scientific publication \u2014 in two research domains by applying the LDA model. The basic input unit for LDA is a set of documents. To organize author information into topics, Rosen-Zvi <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Rosen-ZviTheAuthor04_16-1\" class=\"reference\"><a href=\"#cite_note-Rosen-ZviTheAuthor04-16\" rel=\"external_link\">[16]<\/a><\/sup> and Steyvers <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SteyversProbab04_14-1\" class=\"reference\"><a href=\"#cite_note-SteyversProbab04-14\" rel=\"external_link\">[14]<\/a><\/sup> proposed the author-topic model with different theoretical background. Li <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-LiABiblio15_15-1\" class=\"reference\"><a href=\"#cite_note-LiABiblio15-15\" rel=\"external_link\">[15]<\/a><\/sup> identified the relations between authors and topics by using the author-topic model. They analyzed the topic distribution to examine how many authors are associated with a certain topic. Also through the number of authors, they identified topics that are studied by many researchers. Tang <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TangArnetMiner08_17-1\" class=\"reference\"><a href=\"#cite_note-TangArnetMiner08-17\" rel=\"external_link\">[17]<\/a><\/sup> proposed the ACT model which identifies paper, author, and conference simultaneously. Additionally, they developed the ArnetMiner system for mining academic research social networks. Tang <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TangTopic11_26-0\" class=\"reference\"><a href=\"#cite_note-TangTopic11-26\" rel=\"external_link\">[26]<\/a><\/sup> also supplement ArnetMiner for a topic level expertise search over heterogeneous networks using the ACT model. It generates the most issued topics, author\u2019s interestedness, paper search, academic suggestion, and experts in a specific field. Kim <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-KimExploring16_27-0\" class=\"reference\"><a href=\"#cite_note-KimExploring16-27\" rel=\"external_link\">[27]<\/a><\/sup> adopted the ACT model in terms of citation analysis. They collected their dataset in the field of oncology from PubMed Central, which provides the full-text articles in the biomedical field. They utilized the ACT model for analyzing citation sentences and journals instead of abstracts and conferences.\n<\/p><p>In conclusion, most previous studies identified knowledge structures by adopting not only bibliometric analysis but text-mining techniques such as the LDA model. To supplement bibliometric analysis, there are many attempts to incorporate content analysis into bibliometrics by adopting the LDA model text-mining techniques. However, the main limitation of this application of the LDA model, the representative method for trend analysis, is that it only explains topical trends by using one parameter such as the bag-of-words on documents via topical terms. It is not sufficient to conduct comprehensive analysis for understanding knowledge disciplines. Therefore, in this paper, we apply the ACT model to the bioinformatics field for integrated analysis. Applying the ACT model, we aim to explore the importance of authors and journals in relation to topics. We divided the collected datasets into four periods to trace the changes of topic, author, and journal ranking over time, combining the results with bibliometric analysis.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>In this section, we describe data collection, preprocessing, and keyphrase extraction to feed input into the ACT model. Figure 1 illustrates the overflow of our approach; detailed descriptions of each component are provided in the following section.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Heo_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"48eb4c9c6bbd2e84294fca06295e7b5b\"><img alt=\"Fig1 Heo BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/4\/4c\/Fig1_Heo_BMCBioinformatics2017_18.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Research overflow. Research overflow of our approach consists of data collection, preprocessing, keyphrase extraction, ACT model application, and topic analysis.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Data_collection\">Data collection<\/span><\/h3>\n<p>For analysis, we collected the 48 journals in the bioinformatics field as used by Song and Kim.<sup id=\"rdp-ebb-cite_ref-SongDetect13_11-2\" class=\"reference\"><a href=\"#cite_note-SongDetect13-11\" rel=\"external_link\">[11]<\/a><\/sup> Forty-six out of the 48 journals were found via the advanced search tool provided by PubMed. Two journals, <i>Advanced Bioinformatics<\/i> and <i>Genome Integration<\/i>, were not retrieved from PubMed. We downloaded the 46 PubMed-listed journals in XML format (Table 1). The total number of papers indexed in these journals was 241,569; <i>Biochemistry<\/i> had the greatest number of papers with 62,270, accounting for 25.78% of the collected publications.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 1.<\/b> Statistics of collected publications\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Ranking\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Journal name\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of papers\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Ratio (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Biochemistry<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">62,270\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25.78\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Molecular Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29,968\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.41\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>The EMBO Journal<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17,296\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.16\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Theoretical Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12,200\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.05\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Bioinformatics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9,847\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.08\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Human Molecular Genetics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9,347\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.87\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Genomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8,316\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.44\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>BMC Genomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7,741\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.20\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>BMC Bioinformatics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6,780\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.81\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Protein Science: A publication of the Protein Society<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6,047\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.50\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Proteome Research<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,575\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.31\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Proteomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,545\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.30\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Biotechnology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,204\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.15\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>PLOS Genetics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,139\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.13\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>PLOS Computational Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,852\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.59\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>BMC Research Notes<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,743\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.55\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Mammalian Genome<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,499\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.45\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Genome Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,411\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.41\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>PLOS Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,280\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.36\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Trends in Biochemical Sciences<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,171\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.31\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Trends in Genetics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,035\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.26\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Molecular Modeling<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,852\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.18\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Molecular & cellular proteomics: MCP<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,796\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.16\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Trends in Biotechnology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,353\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.97\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Bulletin of Mathematical Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,331\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.96\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Proteomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,158\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.89\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Physiological Genomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1,794\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.74\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Computer-Aided Molecular Design<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1,706\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.71\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>BMC Systems Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1,397\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.58\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Bioinformation<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1,297\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.54\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Pharmacogenetics and Genomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1,072\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.44\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">32\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Statistical Methods in Medical Research<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">976\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.40\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Computational Neuroscience<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">925\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.38\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Molecular Systems Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">822\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.34\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">35\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Genome Medicine<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">676\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.28\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">36\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Theoretical Biology and Medical Modeling<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">498\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.21\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">37\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Comparative and Functional Genomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">466\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.19\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">38\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Neuroinformatics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">385\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Cancer Informatics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">355\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.15\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Briefings in Functional Genomics & Proteomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">290\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.12\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">41\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Evolutionary Bioinformatics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">249\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">42\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Algorithms for Molecular Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">245\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">43\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Biomedical Semantics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">240\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">44\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>BioData Mining<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">149\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">45\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>EURASIP Journal on Bioinformatics and Systems Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">140\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Source Code for Biology and Medicine<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">131\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\">Total\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">241,569\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.00\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Data_preprocessing_and_keyphrase_extraction\">Data preprocessing and keyphrase extraction<\/span><\/h3>\n<p>We limited the publication year back to 1996 and divided the dataset into the following four time periods to identify the trend of bioinformatics from the birth of the field to present: 1996\u20132000, 2001\u20132005, 2006\u20132010, and 2011\u20132015 (Fig. 2).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Heo_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"94d30b0a450a6f5d8577e89bd85d3c03\"><img alt=\"Fig2 Heo BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/d\/d7\/Fig2_Heo_BMCBioinformatics2017_18.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Research overflow. Data distribution. Publication year of our dataset is from 1996 to 2015. To identify topical trends of bioinformatics, we divided total 20 years into four time periods. X-axis is publication year and Y-axis is the number of papers.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As shown in Fig. 2, there is a relatively consistent increase in the number of papers. There are fewer than half as many papers published in 2015 than in 2014 because we collected our dataset in June 2015. Nevertheless, we included the 2015 data to observe the latest publication trends. \n<\/p><p>Table 2 presents the breakdown of our dataset by period. As in Fig. 2, the fourth period is the most productive, containing 53,520 papers, or 31.46% of the total dataset. The most productive year is 2014, which accounts for 7.20% with 12,251 papers. The total number of papers for all 20 years is 170,099. This number is different from Table 1 (241,569) as a result of preprocessing; we excluded papers that did not have an abstract.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 2.<\/b> Time-based statistics for 20 years\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Year\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of paper\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Ratio (%)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Ranking\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1996\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.36\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1997\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,549\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1998\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,853\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.44\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1999\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,877\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5,947\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Period 1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28,939\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2001\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6,199\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.64\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2002\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6,456\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.80\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2003\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6,668\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.92\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2004\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7,564\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.45\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2005\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8,545\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Period 2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">35,432\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20.83\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2006\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9,845\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.79\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2007\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10,112\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.94\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2008\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10,352\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2009\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10,868\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2010\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11,031\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.49\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Period 3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">52,208\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30.69\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2011\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11,518\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.77\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2012\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11,986\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2013\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11,695\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.88\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2014\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12,251\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2015\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6,070\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.57\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Period 4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">53,520\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31.46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">170,099\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>We extract various metadata, such as the PMID, author, publication year, journal title, title, and abstract from XML formatted records. After XML processing, we combine the title with abstract and conduct keyphrase extraction. For keyphrase extraction, we use MAUI, which has the keyphrase model trained with MeSH terms.<sup id=\"rdp-ebb-cite_ref-MedelyanHuman09_28-0\" class=\"reference\"><a href=\"#cite_note-MedelyanHuman09-28\" rel=\"external_link\">[28]<\/a><\/sup> In this dataset, there are 500 documents and several keys consisting of MeSH terms about each document, manually assigned by the indexer. MAUI is a newer version of the keyphrase extraction algorithm KEA.<sup id=\"rdp-ebb-cite_ref-WittenKEA99_29-0\" class=\"reference\"><a href=\"#cite_note-WittenKEA99-29\" rel=\"external_link\">[29]<\/a><\/sup> Keyphrase extraction enables researchers to select representative phrases to make topic detection more meaningful. Therefore, we use keyphrases extracted from the title or abstract as our input for the ACT model instead of individual words.\n<\/p><p>Table 3 shows the results of keyphrase extraction and other metadata such as the title and publication year from the PubMed record <a class=\"external mw-magiclink-pmid\" rel=\"external_link\" href=\"https:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26030820?dopt=Abstract\" target=\"_blank\">PMID 26030820<\/a>.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 3.<\/b> Example of results of keyphrase extraction and other metadata from PMID of 26030820\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Information\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Content\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Title\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">encoding cell amplitude frequency modulation\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Author\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Micali Gabriele, Aquino Gerardo, Richards David M, Endres Robert G\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Year\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2015\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Journal\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PLOS computational biology\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Keyphrases\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Down-Regulation - Ion Channels - Ions - L Cells (Cell Line) - Ligands - Social Control, Formal - Social Control, Informal - Up-Regulation\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"ACT_model_application\">ACT model application<\/span><\/h3>\n<p>The ACT model, proposed by Tang <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TangArnetMiner08_17-2\" class=\"reference\"><a href=\"#cite_note-TangArnetMiner08-17\" rel=\"external_link\">[17]<\/a><\/sup> as an extension of the LDA model<sup id=\"rdp-ebb-cite_ref-BleiLatent03_25-1\" class=\"reference\"><a href=\"#cite_note-BleiLatent03-25\" rel=\"external_link\">[25]<\/a><\/sup>, is a unified topic model for modeling various metadata simultaneously. This model starts with the assumption that the order of the topic created by the paper, author, and conference is the same. It also estimates the statistical distribution associated with all topics for the purpose of discovering latent topic distribution related with paper, author, and conference. In this paper, two metadata types are changed. First, conference is replaced with journal. Also, a bag-of-keyphrases are used instead of a bag-of-words to represent documents in a more precise manner.\n<\/p><p>Figure 3 illustrates the ACT model, and Table 4 provides a description of the parameters used. Model estimation is conducted by setting parameters, and for estimation of the model parameter, the Gibbs sampling method is employed. Gibbs sampling takes samples from a probability distribution by using the Markov Chain Monte Carlo sampling method. Three parameters for estimating the model are as follows: 1) \u03b8 is the topic probability for a given author (author*topic matrix), 2) \u03c6 is the journal probability for a given topic (topic*journal matrix), and 3) \u03c8 is the word probability for a given topic (topic*word matrix). According to the independence assumption, joint distribution of topic, author, journal, and word stand on the basis A<sub>d<\/sub>, meaning the total number of authors in paper d. In our experiments, we set the hyper-parameters \u03b1, \u03b2, and \u03b3, which are parameters of a prior with \u03b1\u2009=\u200950\/T, \u03b2\u2009=\u20090.01, and \u03b3 = 0.01, respectively. In addition, we fix the number of topics K to 20, the number of top keyphrases to 30, and the number of iterations to 1,000. With these settings, we selected 15 out of 20 topics for analysis.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Heo_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"d6b467be3940b0e28cd4ebd1134f2dff\"><img alt=\"Fig3 Heo BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/2\/2b\/Fig3_Heo_BMCBioinformatics2017_18.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> ACT Model. Author-Conference-Topic (ACT) Model is proposed by Tang <i>et al.<\/i> which is a probabilistic topic model to extract topics, authors, and conference simultaneously. <\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 4.<\/b> Notation and description of the ACT model\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Notation\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Description\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Paper\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">x\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Author\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">w\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Word\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">j\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Journal\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total number of papers\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total number of authors\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">K\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Selected number of topics\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N<sub>d<\/sub>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total number of words in paper d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A<sub>d<\/sub>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Total number of authors in paper d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">z\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Topic\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Author-topic distribution\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03c6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Topic-journal distribution\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03c8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Topic-word distribution\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b1,\u03b2,\u03b3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hyper-parameters of Dirichlet distribution\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Evaluation\">Evaluation<\/span><\/h3>\n<p>To examine the consistency of our results, we repeated each run 10 times with a topic number of 20. After that, we calculated the similarity between topics. For statistical analysis, we computed Pearson correlation coefficients between any two topics and averaged them out. Table 5 shows the average of correlation coefficients per execution. In all runs, Pearson correlation coefficients between topics were weakly, positively correlated. Also, the range of correlation was not wide (0.13 to 0.18). It implies that there was no difference in similarity between topics regardless of different runs. This result can verify consistency and reliability of our topic clusters.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 5.<\/b> Average of Pearson correlation coefficients result\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of runs\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Pearson correlation coefficients\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.155\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.140\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.152\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.177\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.180\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.146\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.136\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.160\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.158\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.178\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In addition, to evaluate the topic model results, we used perplexity which is a well-known measurement in information theory for testing goodness of a model. In our case, we make a test set by collecting bioinformatics journals published in 2016. The sample size is 1,000 papers. In the training set, we divided 20 years into four periods and calculated the perplexity by setting the number of topics as 10, 20, 30, and 50 respectively. The results are presented in Table 6 and Fig. 4. As shown in Table 6 and also confirmed in Fig. 4, there is not much difference in performance in regards to the number of topics by perplexity. However, there is a clear difference among periods by perplexity. In particular, the third period has the highest perplexity value, which implies that it is the most difficult period as to predicting the topic trend in 2016 in the bioinformatics field.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 6.<\/b> Perplexity result of topic model\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Number of topics\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">1996\u20132000\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">2001\u20132005\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">2006\u20132010\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">2011\u20132015\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Average\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,712\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,060\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">875,088\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">501,176\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">345,259\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,978\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3,161\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">726,329\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">513,176\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">311,411\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,872\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,176\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">742,307\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">481,875\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">307,308\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,480\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,149\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">635,960\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">466,676\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">276,816\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Average\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,760\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2,387\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">744,921\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">490,726\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Heo_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"a2326d3998f68364c795275556c1f4fc\"><img alt=\"Fig4 Heo BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/0\/00\/Fig4_Heo_BMCBioinformatics2017_18.gif\" width=\"358\" height=\"234\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Perplexity result. For evaluation of topic modeling results, we used perplexity. We calculated perplexity per each period with the number of topics as 10, 20, 30, and 50. X-axis is period and Y-axis means a perplexity value. <\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Together with this result, we analyzed the results of the ACT model.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>We analyzed leading authors and journals in relation to topics over time. In the following section, we provide the detailed explanations of the trend per period.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Topic_analysis_per_period\">Topic analysis per period<\/span><\/h3>\n<p>The results of our time series topic analysis show that topics seem to be more distinct and subdivided closer to present. In addition, new topics have emerged in recent years, and they do not make a new cluster, which means the exclusive topics become apparent. The results also show that research fields such as molecular biology, genomics, genetics, and proteomics play a supplementary role in biology, but also become diversified into a unique field.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"First_period_analysis\">First period analysis<\/span><\/h4>\n<p>In the first period (1996\u20132000), five dominant topic clusters were identified (Additional file 1: Appendix 1). Those five topics were mainly associated with proteins and peptides. Phrases such as \"molecular biology\" and \"chemical compound\" were widespread, and thermodynamics- and kinematics-related topics appeared. These topics were composed of jargon in their specific fields. The mathematical biology field was represented by topical phrases such as \"database,\" \"cluster analysis,\" \"model,\" \"theoretical,\" and \"software.\"\n<\/p><p>Topics 0, 2, and 3 were about molecular biology, derived from biochemistry and composed of hydrogen bonding\u2013related chemical compounds such as enzymes or lipids. Topics 4, 5, 6, and 7 were related to proteins, peptides, and protein structure. Topics 9 and 14 included words such as \"probability\" and \"statistics,\" which are related to mathematical biology. Topics 13, 17, 18, and 19 covered mutagenesis, disease, and syndromes, which are all related to genetic diseases; mutagenesis consists of gene mutation, and syndromes are caused by genetic disorder. Topic 19 included the word \"genetic,\" a parent category of previously mentioned words. Topics 15 and 16 consisted of kinetics.\n<\/p><p>Protein-related topics were dominant, and authors involved in peptide and protein structure were prevalent in the first period. Authors who were in topic 5, such as A. R. Fersht, J. M. Thornton, C. M. Dobson, L. Serrano, and M. Karplus, had a high probabilistic distribution value, which means they were leading researchers in this area. Their research interest was mainly in protein structure, and they published in the <i>Journal of Molecular Biology<\/i>. This journal appeared in almost all of the topics related to protein and dealt with structure and function of macromolecules, complexes, and protein folding.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Second_period_analysis\">Second period analysis<\/span><\/h4>\n<p>There were four topic clusters and one exclusive topic in the second period (Additional file 1: Appendix 2). In the second period (2001\u20132005), studies about genetics and genomics were actively conducted, and protein-related topics were diversified into subfields such as proteomics. In addition, mathematical biology and computational biology\u2013related topics were maintained in this period.\n<\/p><p>Topics 1, 2, 5, 7, and 11 included DNA mechanisms, molecular structure, genetics, genomics, and diseases caused by DNA or a genome such as Down syndrome, DNA transposable elements, and ribonucleases. Topics 0, 3, 14, and 16 were mainly about proteomics, specifically focusing on protein structure. Topics 12, 18, and 19 addressed biotechnology, molecular modeling, and structure. Topics 8 and 9 focused on mathematical biology and computational biology. Topic 4 exclusively contained enzymology-related phrases such as \"enzyme activators\" and \"oxygen.\" Enzymology-related topics were less common compared with the first period.\n<\/p><p>The second period mainly focused on gene-related topics. Topic 5 had the highest probabilistic distribution among top-ranked authors such as Gregory A. Petsko, L. Aravind, Eugene V. Koonin, Mark Gerstein, and Laurence D. Hurst, who were interested in genomics and biomedical engineering. Those authors published papers in <i>Genome Biology<\/i>, which covered subject matters related to genomics and post-genomics. Similar to the first period, protein-related research was a major topic in the second period. Top-ranked authors in this topic included Ruedi Aebersold, Peter Roepstorff, Pier Giorgio Righetti, Jean-Charles Sanchez, and Peter R. Jungblut. These authors were pioneers of proteomics, and their papers were published in the <i>Journal of Molecular Biology<\/i> and <i>Proteomics<\/i>.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Third_period_analysis\">Third period analysis<\/span><\/h4>\n<p>In the third period (2006\u20132010), topics were divided into three clusters: genomics, proteomics, and other (Additional file 1: Appendix 3). Different from the first two periods, four exclusive topics existed and seemed to be distinct from topics in the other three periods. For instance, studies about genomics or proteomics were more diversified than in the earlier periods. Exclusive topics that were not included in two large fields emerged, indicating that bioinformatics research was conducted in various fields related to bioinformatics.\n<\/p><p>Topics 3, 7, 10, 11, 13, and 16 consisted of proteomics, protein evolution, and protein structure. Proteomics-related topics were subdivided. The representative journals in the area were <i>Proteomics<\/i>, the '<i>Journal of Proteome Research<\/i>, and the <i>Journal of Proteomics<\/i>. Topics 5, 6, 12, 14, and 19 were gene-related topics such as gene expression, gene transcription, and genomics. Gene-related studies became prevalent in the second period. The distinct topics that appeared in the third period were topics 0, 15, 17, and 18. Topic 0, molecular biology, especially focused on hydrogen bonding. In the first and second periods, topic 15 included various topics related to theoretical biology. Topic 17 addressed hepatitis, the infection in liver cells and tissues. Different from previous periods, in the third period topics were associated with specific diseases. Topic 18 included peptide-associated phrases, and, unlike prior periods, concrete themes like specific chemical compounds and protein appeared.\n<\/p><p>Overall, protein-related topics were most common in the third period. The third period also has more sub-divided and distinct topics than previous periods did. In this period, general topics such as proteomics appeared, as did specific topics such as protein evolution, protein analytics, and protein ubiquitin. Among these areas, the topic with the highest distribution was protein analytics, and it was sub-categorized in proteomics. Top-ranked authors in this period included Matthias Mann, Ruedi Aebersold, Richard D. Smith, Albert J. R. Heck, and Visith Thongboonkerd. They were experts in protein analytics and commonly used mass spectrometry for their analyses. They actively published in the <i>Journal of Proteome Research<\/i> and <i>Proteomics<\/i>. These two journals were top-rated in protein-related topics. The <i>Journal of Proteome Research<\/i> was computer technology\u2013oriented and focused on protein-analysis research. The journal with the highest probabilistic distribution in all topic areas was the <i>EMBO Journal<\/i>. This journal focused on molecular biology and also covered proteomics.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Fourth_period_analysis\">Fourth period analysis<\/span><\/h4>\n<p>The fourth period (2011\u20132015) showed three major topic clusters and two exclusive topics (Additional file 1: Appendix 4). Similar to the third period, the topics related with genomics and proteomics were further divided into subfields and represented concrete topical characteristics. Compared with the third period\u2019s results, theoretical biology\u2013related topics formed one cluster. The compositions of the cluster were one big topic (systems biology) and four sub-divided topics.\n<\/p><p>Topics 1 and 16 were theoretical biology\u2013related, and topics 6 and 10 were about systems biology. They could be clustered as a broader category of system biology. The representative journals in this cluster were <i>PLOS Computational Biology<\/i>, <i>Journal of Theoretical Biology<\/i>, and <i>Journal of Computational Neuroscience<\/i>, which were focused on systems biology. Topics 0, 11, 12, 18, and 19 were about genetics and genomics. Topics 4, 9, 13, and 17 represented proteomics. Topics 8 and 15 were exckusive, each of which was related to molecular biology and cell biology. Topic 8 included phrases like \"hydrogen bonding\" and \"GTP-binding proteins,\" and topic 15 contained phrases like \"enteroendocrine cells\" and \"COS cells.\" The top journals in these areas were <i>Biochemistry<\/i>, the <i>Journal of Molecular Biology<\/i>, and the <i>Journal of Molecular Modeling<\/i>.\n<\/p><p>In the fourth period, the major topics were systems biology, genomics, and proteomics. Topics that were not in the main stream of bioinformatics were found in this period, and topics about theoretical biology and systems biology become a distinct cluster. This means that these areas were growing in the bioinformatics area. The representative researchers in this area were Martin A. Nowak, Yoh Iwasa, Mike Steel, Ulf Dieckmann, and Liam Paninski. They were mostly involved in mathematics and theoretical biology. The journal which had the highest probabilistic distribution was the <i>Journal of Theoretical Biology<\/i>. This journal focused on research that combines biology and topics such as statistical analysis, mathematical definition, comparative research, experiments, and computer simulation. The second raked journal was the <i>Journal of Bioinformatics<\/i>, which mainly accepted research about genome bioinformatics and computational biology.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>In this section, we analyze the results from three different perspectives: topical keyphrase, journal, and author. In addition, to further identify which researchers and journals focus on which topic over time, the results of the ACT model (top-ranked keyphrases, authors, and journals) are examined in an integrated perspective.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Time_series_analysis\">Time series analysis<\/span><\/h3>\n<p>One interesting observation is that keyphrases related with genes or genetic processes such as \"gene expression,\" \"down-regulation,\" and \"up-regulation\" were not ranked high in the first period. However, they emerged as top keyphrases in later periods. In particular, \"proteome,\" \"reproducibility of results,\" \"proteomics,\" and \"genotype\" did not appear in the first period but emerged gradually after that. In addition, the term \"algorithms\" appeared most frequently in the fourth period. In author perspective, across the four periods, the number of unique authors was 1,396. Top ranked author Richard Robinson appeared in five topics. Seven authors, including Liza Gross, appeared four times, 45 authors appeared three times, 137 authors appeared twice, and the remaining 1,184 authors were shown only in one topic. There was no author who appeared in all four periods. Thirty-nine authors appeared across three periods, 125 authors appeared in two periods, and 1,210 authors appeared only in a period. In journal centered view, only 21 out of 46 journals appeared in the first period. In the second and third period, 34 and 46 journals were presented respectively. Forty-five journals appeared in the last period; one the journals, <i>Briefings in Functional Genomics & Proteomics<\/i>, did not appear in the last period.\n<\/p><p>These results imply that the bioinformatics field is diversified and new topical disciplines have recently emerged. For instance, proteomics-related topics started to appear in the second period, became segmented into detailed research fields and later evolved in the third and fourth periods. In addition, while conceptual biology\u2013related topics existed in the first period, they became clearly progressed in the fourth period. Conversely, the topics about kinetics appeared in the first period but then faded out.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Integrated_view_of_graph_pattern_analysis\">Integrated view of graph pattern analysis<\/span><\/h3>\n<p>For further integrated analysis, we examined top journals with their topical probability in all four periods. We also checked the authors and topical keyphrases which were topically matched with the journals. We identified that there were four different patterns in a journal\u2019s topical distribution such as rising, falling, concave, and convex patterns. In Fig. 5, we only presented graphs which drastically changed in terms of the probability value of topics between periods. Additionally in each graph, we presented the top five ranked authors and keyphrases which had a high probability value across four periods.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Heo_BMCBioinformatics2017_18.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"842ff84474d097672198cc0e2036881d\"><img alt=\"Fig5 Heo BMCBioinformatics2017 18.gif\" src=\"https:\/\/www.limswiki.org\/images\/c\/cd\/Fig5_Heo_BMCBioinformatics2017_18.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Journal focused topic distribution with related authors and keyphrases. For integrated pattern analysis, we examined eight representative journals with top authors and keyphrases. Patterns were classified as four outstanding ones such as rising (a-b), falling (c-f), convex \ufeff(g) and concave (h).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>We explained four outstanding cases in each pattern and made a list of journals which are shown in all four periods (Table 7). First for the rising pattern, the journal <i>BMC Bioinformatics<\/i> had 0.86060 gaps between maximum and minimum probability. It was the highest gap by probability in all rising patterns. The average impact factor of this journal provided by journal citation report (JCR) was 3.0806 in 2015. In this context, <i>BMC Bioinformatics<\/i> could be regarded as the most promising journal in the bioinformatics field. The journal has grown steadily through 20 years. The authors belonging to the similar topical scope of <i>BMC Bioinformatics<\/i> were presented in graph (a). The top ranked authors shared common characteristics: the authors were majored in computer science or statistics and then applied the technique to biomedical or biology later on, and their common research interests were bioinformatics or biostatistics. As shown in graph (a), the topical keyphrases related with the journal were not focused only on biology research fields. The word \"algorithms\" represents the informatics field, while \"genome,\" \"genomics,\" and \"gtp-binding proteins\" relate to the genomics fields, and the word \"protein s\" represents protein- or gene-related scopes. The scope of the journal is in computational and statistical methods for the modeling and analysis of all kinds of biological data, as well as other areas of computational biology. The results indicated that the publication trend of <i>BMC Bioinformatics<\/i> changed to genetics or genomics converged with informatics. \n<\/p><p>In the case of the falling pattern of the journal <i>Biochemistry<\/i>, as shown in graph (c), the journal had 4.29181 gap between the maximum and minimum value. The average impact factor of this journal in four periods was 3.75322. The journal had a somewhat high probability value, but the impact factor in each period decreased gradually (e.g., first period: 4.4785 to fourth period: 3.1768). This decreasing pattern implied that in bioinformatics field, the journal dealt mainly with biochemistry, biophysical chemistry, and molecular biology, but it had not converged with informatics. That means the biochemistry field did not have much interaction with informatics fields. \n<\/p><p>The top ranked authors shown in graph (c) were celebrated scholars in the biochemistry field. Commonly, they all researched only the biochemistry field, and we could not find that they had a connection with the informatics fields. In the same context, top ranked keyphrases of the journal were mostly related to biochemistry research fields. The informatics-related keyphrase did not appear in the top word list.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 7.<\/b> The list of journals showed in all periods\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Journal name\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Sum of probability\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Average of probability\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Biochemistry<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13.13233\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.28308\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Bioinformatics<\/i> (Oxford, England)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.49214\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62304\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>BMC Bioinformatics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.64624\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.41156\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>BMC Genomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.92022\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.48005\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Bulletin of Mathematical Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.68576\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.17144\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Genome Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.28690\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.32173\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Genomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.40453\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.60113\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Human Molecular Genetics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.98031\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.74508\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Biotechnology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.32392\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.33098\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Computational Neuroscience<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.42877\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10719\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Computer-Aided Molecular Design<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.60322\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.15081\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Molecular Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.82534\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.45633\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Journal of Theoretical Biology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.00820\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.75205\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Mammalian Genome<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.06714\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.26678\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Physiological Genomics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.61599\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.15400\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Protein Science<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.16372\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.54093\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Statistical Methods in Medical Research<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.25062\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06266\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>The EMBO Journal<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.35207\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.83802\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Trends in Biochemical Sciences<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.50571\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.12643\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Trends in Biotechnology<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.50871\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.12718\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>Trends in Genetics<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.73352\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.18338\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Not only the rising and falling patterns, but the convex and concave patterns of journals exist. In graph (g), <i>BMC Genomics<\/i> was the journal showing the convex pattern. <i>BMC Genomics<\/i> dealt with genome-scale analysis, functional genomics, and proteomics. The average impact factor from 2000 to 2015 was 4.0464. The topical probability of the journal rose from the first period (0.00125) to the third period (1.19977), but since then the probability dropped to 0.51576. The gap between maximum and minimum value was 1.19852. The journal changed the status to open-access in 2000, and maybe it caused a slight rise from first period to second period. For the drastic increase from second period to third period, the number of total citations increased approximately two times according to the JCR. The increasing and decreasing number in total citations may cause the wave in graph (g). The top five ranked authors\u2019 research interests had various characteristics. N. A. Jenkins and N.G. Copeland were a couple, with Jenkins interested in molecular and cellular biology, while the husband Copeland was in the biochemistry field. They collaborated a lot and both were celebrated researchers in the field. The other researchers also had different research areas, including treatment or preventive therapy (Gregory A. Petsko), genomic medicine (Y. Nakamura), and genome analysis (L. Aravind). Through all four periods, the journal mainly published articles which dealt with the genomic related keyphrases. The authors and keyphrases located in this journal were not quite related to the informatics fields. \n<\/p><p>Last, the concave pattern was shown in graph (h), and <i>Human Molecular Genetics<\/i> was the representative journal. The journal had steadily decreasing impact factor flow in 20 years (e.g., first period: 9.05475 to fourth period: 6.8766). The gap between maximum and minimum probability was 0.81973. In graph (h), the point of drastic decrease can be readily observed. Top ranked authors related with the journal focused on genetics. N. G. Copeland and N. A. Jenkins appeared again. The rest of the authors did not overlap in research areas. The journal was interested in broad genetics-related topical keyphrases but not related with informatics fields.\n<\/p><p>Through the pattern analysis integrated with journal, author, and keyphrase, we identified that the field of bioinformatics was a converging area, and certain journals clearly showed rising and falling patterns. Different from period analysis, the integrated view of analysis showed journals\u2019 topical trends over time along with top journals and researchers.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Bioinformatics mainly tackles biological problems at the molecular level using applied mathematics, information science, statistics, computer science, chemistry, and biochemistry. This characteristic of bioinformatics has driven the field to become interdisciplinary, combining approaches from various fields to make use of a large amount of data.\n<\/p><p>In this study, we investigated the bioinformatics field using the ACT model to conduct comprehensive topic analyses of keyphrases, authors, and journals. To this end, we collected 46 journals belonging to the bioinformatics field by searching journal name in PubMed, yielding 170,099 papers. To analyze topic evolution over time, we divided the collected datasets into four, five-year periods: 1996\u20132000, 2001\u20132005, 2006\u20132010, and 2011\u20132015. In a time series topic analysis, we examined topic clusters within periods. In more recent periods, distinct characteristics of the field emerged and more new topics were presented independently. In addition, we analyzed trends in keyphrases, authors, and journals. Our keyphrase analysis similarly indicated the emergence of greater interdisciplinary research over time. In our author analysis, we observed the pattern of authors who appear in top rank. In our journal analysis, we analyzed the common topic area of top journals and identified major focus areas of those top journals, including computational biology, theoretical biology, and mathematical biology. Also, we examined topic distribution over journals with top ranked authors and keyphrases. In this analysis, journals were identified as four patterns over time such as rising, falling, convex, and concave patterns.\n<\/p><p>The results of these analyses imply that the bioinformatics field is highly interdisciplinary, consisting of active convergence studies. In addition, we observed that the characteristics of the bioinformatics field become more distinct and more specific, and the supporting role of peripheral fields of bioinformatics, such as conceptual, mathematical, and systems biology, gradually increases over time, though the core fields of proteomics, genomics, and genetics are still the major topics. This is consistently confirmed by analysis of topic distribution of journals over time as well as integrated analysis of top ranked keyphrases, authors, and journals.\n<\/p><p>In the future, we plan to apply the same approach to other domains, such as information science. We also plan to use other metadata such as MeSH terms and bio-entities to compare with keyphrases. In addition, we plan to explore how to infer authors\u2019 interests by time series analysis and identify representative authors and the journals that are the best suited for a paper on a particular subject.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Declarations\">Declarations<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h3>\n<p>Not applicable.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h4>\n<p>This work was supported by the Bio-Synergy Research Project (NRF-2013M3A9C4078138) of the Ministry of Science, ICT and Future Planning through the National Research Foundation. This work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. NRF-2014R1A2A2A01004454). The publication charges for this article was funded by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Availability_of_data_and_materials\">Availability of data and materials<\/span><\/h4>\n<p>Not applicable.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Authors.27_contributions\">Authors' contributions<\/span><\/h4>\n<p>GEH conceived and designed the study. GEH and KYK conducted the analysis and drafted the manuscript. MS designed and coordinated the study. All authors contributed to the writing of the manuscript. All authors read and approved the final manuscript.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h4>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Consent_for_publication\">Consent for publication<\/span><\/h4>\n<p>Not applicable.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Ethics_approval_and_consent_to_participate\">Ethics approval and consent to participate<\/span><\/h4>\n<p>Not applicable.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"About_this_supplement\">About this supplement<\/span><\/h4>\n<p>This article has been published as part of BMC Bioinformatics Volume 18 Supplement 7, 2017: Proceedings of the Tenth International Workshop on Data and Text Mining in Biomedical Informatics. The full contents of the supplement are available online at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/supplements\/volume-18-supplement-7\" target=\"_blank\">https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/supplements\/volume-18-supplement-7<\/a>.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Publisher.E2.80.99s_note\">Publisher\u2019s note<\/span><\/h4>\n<p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Additional_files\">Additional files<\/span><\/h2>\n<ul><li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1186%2Fs12859-017-1640-x\/MediaObjects\/12859_2017_1640_MOESM1_ESM.docx\" target=\"_blank\">Additional File 1<\/a>: Appendix 1. ACT model results in first period. Appendix 2. ACT model results in second period. Appendix 3. ACT model results in third period. Appendix 4. ACT model results in fourth period. (DOCX 48 kb)<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-DongPublic15-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DongPublic15_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dong, D.; Chen, M.-L. (2015). \"Publication trends and co-citation mapping of translation studies between 2000 and 2015\". <i>Scientometrics<\/i> <b>105<\/b> (2): 1111\u20131128. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11192-015-1769-1\" target=\"_blank\">10.1007\/s11192-015-1769-1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Publication+trends+and+co-citation+mapping+of+translation+studies+between+2000+and+2015&rft.jtitle=Scientometrics&rft.aulast=Dong%2C+D.%3B+Chen%2C+M.-L.&rft.au=Dong%2C+D.%3B+Chen%2C+M.-L.&rft.date=2015&rft.volume=105&rft.issue=2&rft.pages=1111%E2%80%931128&rft_id=info:doi\/10.1007%2Fs11192-015-1769-1&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChenAlzh14-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ChenAlzh14_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-ChenAlzh14_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chen, H.; Wan, Y.; Jiang, S.; Cheng, Y. (2014). \"Alzheimer\u2019s disease research in the future: bibliometric analysis of cholinesterase inhibitors from 1993 to 2012\". <i>Scientometrics<\/i> <b>98<\/b> (3): 1865\u20131877. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11192-013-1132-3\" target=\"_blank\">10.1007\/s11192-013-1132-3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Alzheimer%E2%80%99s+disease+research+in+the+future%3A+bibliometric+analysis+of+cholinesterase+inhibitors+from+1993+to+2012&rft.jtitle=Scientometrics&rft.aulast=Chen%2C+H.%3B+Wan%2C+Y.%3B+Jiang%2C+S.%3B+Cheng%2C+Y.&rft.au=Chen%2C+H.%3B+Wan%2C+Y.%3B+Jiang%2C+S.%3B+Cheng%2C+Y.&rft.date=2014&rft.volume=98&rft.issue=3&rft.pages=1865%E2%80%931877&rft_id=info:doi\/10.1007%2Fs11192-013-1132-3&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SoteriadesABiblio06-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SoteriadesABiblio06_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-SoteriadesABiblio06_3-1\" rel=\"external_link\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Soteriades, E.S.; Falagas, M.E. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1766930\" target=\"_blank\">\"Alzheimer\u2019s disease research in the future: Bibliometric analysis of cholinesterase inhibitors from 1993 to 2012\"<\/a>. <i>BMC Public Health<\/i> <b>6<\/b>: 301. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2458-6-301\" target=\"_blank\">10.1186\/1471-2458-6-301<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1766930\/\" target=\"_blank\">PMC1766930<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17173665\" target=\"_blank\">17173665<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1766930\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1766930<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Alzheimer%E2%80%99s+disease+research+in+the+future%3A+Bibliometric+analysis+of+cholinesterase+inhibitors+from+1993+to+2012&rft.jtitle=BMC+Public+Health&rft.aulast=Soteriades%2C+E.S.%3B+Falagas%2C+M.E.&rft.au=Soteriades%2C+E.S.%3B+Falagas%2C+M.E.&rft.date=2006&rft.volume=6&rft.pages=301&rft_id=info:doi\/10.1186%2F1471-2458-6-301&rft_id=info:pmc\/PMC1766930&rft_id=info:pmid\/17173665&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1766930&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UgoliniABiblio07-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-UgoliniABiblio07_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-UgoliniABiblio07_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ugolini, D.; Puntoni, R.; Perera, F.P. et al. (2007). \"A bibliometric analysis of scientific production in cancer molecular epidemiology\". <i>Carcinogenesis<\/i> <b>28<\/b>: 8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fcarcin%2Fbgm129\" target=\"_blank\">10.1093\/carcin\/bgm129<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17548902\" target=\"_blank\">17548902<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+bibliometric+analysis+of+scientific+production+in+cancer+molecular+epidemiology&rft.jtitle=Carcinogenesis&rft.aulast=Ugolini%2C+D.%3B+Puntoni%2C+R.%3B+Perera%2C+F.P.+et+al.&rft.au=Ugolini%2C+D.%3B+Puntoni%2C+R.%3B+Perera%2C+F.P.+et+al.&rft.date=2007&rft.volume=28&rft.pages=8&rft_id=info:doi\/10.1093%2Fcarcin%2Fbgm129&rft_id=info:pmid\/17548902&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangABiblio15-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangABiblio15_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, L.; Chen, X.; Bao, A. et al. (2015). \"A bibliometric analysis of research on Central Asia during 1990\u20132014\". <i>Scientometrics<\/i> <b>105<\/b> (2): 1223\u20131237. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11192-015-1727-y\" target=\"_blank\">10.1007\/s11192-015-1727-y<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+bibliometric+analysis+of+research+on+Central+Asia+during+1990%E2%80%932014&rft.jtitle=Scientometrics&rft.aulast=Wang%2C+L.%3B+Chen%2C+X.%3B+Bao%2C+A.+et+al.&rft.au=Wang%2C+L.%3B+Chen%2C+X.%3B+Bao%2C+A.+et+al.&rft.date=2015&rft.volume=105&rft.issue=2&rft.pages=1223%E2%80%931237&rft_id=info:doi\/10.1007%2Fs11192-015-1727-y&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BornmannGrowth15-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BornmannGrowth15_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-BornmannGrowth15_6-1\" rel=\"external_link\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bornmann, L.; Mutz, R. (2015). \"Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references\". <i>Journal of the Association for Information Science and Technology<\/i> <b>66<\/b> (11): 2215\u20132222. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fasi.23329\" target=\"_blank\">10.1002\/asi.23329<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Growth+rates+of+modern+science%3A+A+bibliometric+analysis+based+on+the+number+of+publications+and+cited+references&rft.jtitle=Journal+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Bornmann%2C+L.%3B+Mutz%2C+R.&rft.au=Bornmann%2C+L.%3B+Mutz%2C+R.&rft.date=2015&rft.volume=66&rft.issue=11&rft.pages=2215%E2%80%932222&rft_id=info:doi\/10.1002%2Fasi.23329&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeaneyType2Dia15-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GeaneyType2Dia15_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-GeaneyType2Dia15_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Geaney, F.; Scutaru, C.; Kelly, C. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4514795\" target=\"_blank\">\"Type 2 Diabetes Research Yield, 1951-2012: Bibliometrics Analysis and Density-Equalizing Mapping\"<\/a>. <i>PLoS One<\/i> <b>10<\/b>: 7. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0133009\" target=\"_blank\">10.1371\/journal.pone.0133009<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4514795\/\" target=\"_blank\">PMC4514795<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26208117\" target=\"_blank\">26208117<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4514795\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4514795<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Type+2+Diabetes+Research+Yield%2C+1951-2012%3A+Bibliometrics+Analysis+and+Density-Equalizing+Mapping&rft.jtitle=PLoS+One&rft.aulast=Geaney%2C+F.%3B+Scutaru%2C+C.%3B+Kelly%2C+C.+et+al.&rft.au=Geaney%2C+F.%3B+Scutaru%2C+C.%3B+Kelly%2C+C.+et+al.&rft.date=2015&rft.volume=10&rft.pages=7&rft_id=info:doi\/10.1371%2Fjournal.pone.0133009&rft_id=info:pmc\/PMC4514795&rft_id=info:pmid\/26208117&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4514795&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mac.C3.ADas-ChapulaBiblio02-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Mac.C3.ADas-ChapulaBiblio02_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-Mac.C3.ADas-ChapulaBiblio02_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mac\u00edas-Chapula, C.A.; Mijangos-Nolasco, A. (2002). \"Bibliometric analysis of AIDS literature in Central Africa\". <i>Scientometrics<\/i> <b>54<\/b> (2): 309\u2013317. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1023%2FA%3A1016074230843\" target=\"_blank\">10.1023\/A:1016074230843<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bibliometric+analysis+of+AIDS+literature+in+Central+Africa&rft.jtitle=Scientometrics&rft.aulast=Mac%C3%ADas-Chapula%2C+C.A.%3B+Mijangos-Nolasco%2C+A.&rft.au=Mac%C3%ADas-Chapula%2C+C.A.%3B+Mijangos-Nolasco%2C+A.&rft.date=2002&rft.volume=54&rft.issue=2&rft.pages=309%E2%80%93317&rft_id=info:doi\/10.1023%2FA%3A1016074230843&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SeglenScientific00-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SeglenScientific00_9-0\" rel=\"external_link\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-SeglenScientific00_9-1\" rel=\"external_link\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Seglen, P.O.; Aksnes, D.W. (2000). \"Scientific Productivity and Group Size: A Bibliometric Analysis of Norwegian Microbiological Research\". <i>Scientometrics<\/i> <b>49<\/b> (1): 125\u2013143. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1023%2FA%3A1005665309719\" target=\"_blank\">10.1023\/A:1005665309719<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Scientific+Productivity+and+Group+Size%3A+A+Bibliometric+Analysis+of+Norwegian+Microbiological+Research&rft.jtitle=Scientometrics&rft.aulast=Seglen%2C+P.O.%3B+Aksnes%2C+D.W.&rft.au=Seglen%2C+P.O.%3B+Aksnes%2C+D.W.&rft.date=2000&rft.volume=49&rft.issue=1&rft.pages=125%E2%80%93143&rft_id=info:doi\/10.1023%2FA%3A1005665309719&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JeongTimeGap14-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JeongTimeGap14_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-JeongTimeGap14_10-1\" rel=\"external_link\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jeong, D.-H.; Song, M. (2014). \"Time gap analysis by the topic model-based temporal technique\". <i>Journal of Infometrics<\/i> <b>8<\/b> (3): 776\u2013790. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.joi.2014.07.005\" target=\"_blank\">10.1016\/j.joi.2014.07.005<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Time+gap+analysis+by+the+topic+model-based+temporal+technique&rft.jtitle=Journal+of+Infometrics&rft.aulast=Jeong%2C+D.-H.%3B+Song%2C+M.&rft.au=Jeong%2C+D.-H.%3B+Song%2C+M.&rft.date=2014&rft.volume=8&rft.issue=3&rft.pages=776%E2%80%93790&rft_id=info:doi\/10.1016%2Fj.joi.2014.07.005&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SongDetect13-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SongDetect13_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-SongDetect13_11-1\" rel=\"external_link\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-SongDetect13_11-2\" rel=\"external_link\">11.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Song, M.; Kim, S.Y. (2013). \"Detecting the knowledge structure of bioinformatics by mining full-text collections\". <i>Scientometrics<\/i> <b>96<\/b> (1): 183\u2013201. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11192-012-0900-9\" target=\"_blank\">10.1007\/s11192-012-0900-9<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Detecting+the+knowledge+structure+of+bioinformatics+by+mining+full-text+collections&rft.jtitle=Scientometrics&rft.aulast=Song%2C+M.%3B+Kim%2C+S.Y.&rft.au=Song%2C+M.%3B+Kim%2C+S.Y.&rft.date=2013&rft.volume=96&rft.issue=1&rft.pages=183%E2%80%93201&rft_id=info:doi\/10.1007%2Fs11192-012-0900-9&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SongProduct13-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SongProduct13_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-SongProduct13_12-1\" rel=\"external_link\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Song, M.; Kim, S.Y.; Zhang, G. et al. (2014). \"Productivity and influence in bioinformatics: A bibliometric analysis using PubMed central\". <i>Journal of the Association for Information Science and Technology<\/i> <b>65<\/b> (2): 352\u2013371. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fasi.22970\" target=\"_blank\">10.1002\/asi.22970<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Productivity+and+influence+in+bioinformatics%3A+A+bibliometric+analysis+using+PubMed+central&rft.jtitle=Journal+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Song%2C+M.%3B+Kim%2C+S.Y.%3B+Zhang%2C+G.+et+al.&rft.au=Song%2C+M.%3B+Kim%2C+S.Y.%3B+Zhang%2C+G.+et+al.&rft.date=2014&rft.volume=65&rft.issue=2&rft.pages=352%E2%80%93371&rft_id=info:doi\/10.1002%2Fasi.22970&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YanResearch15-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-YanResearch15_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-YanResearch15_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yan, E. (2015). \"Research dynamics, impact, and dissemination: A topic-level analysis\". <i>Journal of the Association for Information Science and Technology<\/i> <b>66<\/b> (1): 2357\u20132372. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fasi.23324\" target=\"_blank\">10.1002\/asi.23324<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Research+dynamics%2C+impact%2C+and+dissemination%3A+A+topic-level+analysis&rft.jtitle=Journal+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Yan%2C+E.&rft.au=Yan%2C+E.&rft.date=2015&rft.volume=66&rft.issue=1&rft.pages=2357%E2%80%932372&rft_id=info:doi\/10.1002%2Fasi.23324&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SteyversProbab04-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SteyversProbab04_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-SteyversProbab04_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Steyvers, M.; Smyth, P.; Rosen-Zvi, M.; Griffiths, T. (2004). \"Probabilistic author-topic models for information discovery\". <i>Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining<\/i> <b>2004<\/b>: 306\u2013315. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1014052.1014087\" target=\"_blank\">10.1145\/1014052.1014087<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Probabilistic+author-topic+models+for+information+discovery&rft.jtitle=Proceedings+of+the+Tenth+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining&rft.aulast=Steyvers%2C+M.%3B+Smyth%2C+P.%3B+Rosen-Zvi%2C+M.%3B+Griffiths%2C+T.&rft.au=Steyvers%2C+M.%3B+Smyth%2C+P.%3B+Rosen-Zvi%2C+M.%3B+Griffiths%2C+T.&rft.date=2004&rft.volume=2004&rft.pages=306%E2%80%93315&rft_id=info:doi\/10.1145%2F1014052.1014087&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiABiblio15-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiABiblio15_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiABiblio15_15-1\" rel=\"external_link\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, D.; Okamoto, J.; Liu, H.; Leischow, S. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4432889\" target=\"_blank\">\"A bibliometric analysis on tobacco regulation investigators\"<\/a>. <i>BioData Mining<\/i> <b>8<\/b>: 11. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13040-015-0043-7\" target=\"_blank\">10.1186\/s13040-015-0043-7<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4432889\/\" target=\"_blank\">PMC4432889<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25984237\" target=\"_blank\">25984237<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4432889\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4432889<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+bibliometric+analysis+on+tobacco+regulation+investigators&rft.jtitle=BioData+Mining&rft.aulast=Li%2C+D.%3B+Okamoto%2C+J.%3B+Liu%2C+H.%3B+Leischow%2C+S.&rft.au=Li%2C+D.%3B+Okamoto%2C+J.%3B+Liu%2C+H.%3B+Leischow%2C+S.&rft.date=2015&rft.volume=8&rft.pages=11&rft_id=info:doi\/10.1186%2Fs13040-015-0043-7&rft_id=info:pmc\/PMC4432889&rft_id=info:pmid\/25984237&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4432889&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Rosen-ZviTheAuthor04-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Rosen-ZviTheAuthor04_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-Rosen-ZviTheAuthor04_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rosen-Zvi, M.; Griffiths, T., Steyvers, M.; Smyth, P. (2004). \"The author-topic model for authors and documents\". <i>Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence<\/i> <b>2004<\/b>: 487\u2013494. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 0974903906.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+author-topic+model+for+authors+and+documents&rft.jtitle=Proceedings+of+the+20th+Conference+in+Uncertainty+in+Artificial+Intelligence&rft.aulast=Rosen-Zvi%2C+M.%3B+Griffiths%2C+T.%2C+Steyvers%2C+M.%3B+Smyth%2C+P.&rft.au=Rosen-Zvi%2C+M.%3B+Griffiths%2C+T.%2C+Steyvers%2C+M.%3B+Smyth%2C+P.&rft.date=2004&rft.volume=2004&rft.pages=487%E2%80%93494&rft.isbn=0974903906&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TangArnetMiner08-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TangArnetMiner08_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-TangArnetMiner08_17-1\" rel=\"external_link\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-TangArnetMiner08_17-2\" rel=\"external_link\">17.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tang, J.; Zhang, J.; Yao, L. et al. (2008). \"ArnetMiner: Extraction and mining of academic social networks\". <i>Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining<\/i> <b>2008<\/b>: 990-998. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1401890.1402008\" target=\"_blank\">10.1145\/1401890.1402008<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ArnetMiner%3A+Extraction+and+mining+of+academic+social+networks&rft.jtitle=Proceedings+of+the+14th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining&rft.aulast=Tang%2C+J.%3B+Zhang%2C+J.%3B+Yao%2C+L.+et+al.&rft.au=Tang%2C+J.%3B+Zhang%2C+J.%3B+Yao%2C+L.+et+al.&rft.date=2008&rft.volume=2008&rft.pages=990-998&rft_id=info:doi\/10.1145%2F1401890.1402008&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RamosABiblio08-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RamosABiblio08_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ramos, J.M.; Padilla, S.; Masi\u00e1, M.; Guti\u00e9rrez, F. (2008). \"A bibliometric analysis of tuberculosis research indexed in PubMed, 1997-2006\". <i>International Journal of Tuberculosis and Lung Disease<\/i> <b>2008<\/b>: 1461\u20138. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19017458\" target=\"_blank\">19017458<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+bibliometric+analysis+of+tuberculosis+research+indexed+in+PubMed%2C+1997-2006&rft.jtitle=International+Journal+of+Tuberculosis+and+Lung+Disease&rft.aulast=Ramos%2C+J.M.%3B+Padilla%2C+S.%3B+Masi%C3%A1%2C+M.%3B+Guti%C3%A9rrez%2C+F.&rft.au=Ramos%2C+J.M.%3B+Padilla%2C+S.%3B+Masi%C3%A1%2C+M.%3B+Guti%C3%A9rrez%2C+F.&rft.date=2008&rft.volume=2008&rft.pages=1461%E2%80%938&rft_id=info:pmid\/19017458&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ClaudeBiblio04-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ClaudeBiblio04_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Claude, R.; Charles-Daniel, A.; Jean, A.; Jean-Francois, G. (2004). \"Bibliometric overview of the utilization of artificial neural networks in medicine and biology\". <i>Scientometrics<\/i> <b>59<\/b> (1): 117\u2013130. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1023%2FB%3ASCIE.0000013302.59845.34\" target=\"_blank\">10.1023\/B:SCIE.0000013302.59845.34<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bibliometric+overview+of+the+utilization+of+artificial+neural+networks+in+medicine+and+biology&rft.jtitle=Scientometrics&rft.aulast=Claude%2C+R.%3B+Charles-Daniel%2C+A.%3B+Jean%2C+A.%3B+Jean-Francois%2C+G.&rft.au=Claude%2C+R.%3B+Charles-Daniel%2C+A.%3B+Jean%2C+A.%3B+Jean-Francois%2C+G.&rft.date=2004&rft.volume=59&rft.issue=1&rft.pages=117%E2%80%93130&rft_id=info:doi\/10.1023%2FB%3ASCIE.0000013302.59845.34&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PatraBiblio06-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PatraBiblio06_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Patra, S.K.; Mishra, S. (2006). \"Bibliometric study of bioinformatics literature\". <i>Scientometrics<\/i> <b>67<\/b> (3): 477\u2013489. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1556%2FScient.67.2006.3.9\" target=\"_blank\">10.1556\/Scient.67.2006.3.9<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bibliometric+study+of+bioinformatics+literature&rft.jtitle=Scientometrics&rft.aulast=Patra%2C+S.K.%3B+Mishra%2C+S.&rft.au=Patra%2C+S.K.%3B+Mishra%2C+S.&rft.date=2006&rft.volume=67&rft.issue=3&rft.pages=477%E2%80%93489&rft_id=info:doi\/10.1556%2FScient.67.2006.3.9&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SongAnal14-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SongAnal14_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Song, M.; Heo, G.E.; Kim, S.Y. (2014). \"Analyzing topic evolution in bioinformatics: Investigation of dynamics of the field with conference data in DBLP\". <i>Scientometrics<\/i> <b>101<\/b> (1): 397\u2013428. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11192-014-1246-2\" target=\"_blank\">10.1007\/s11192-014-1246-2<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Analyzing+topic+evolution+in+bioinformatics%3A+Investigation+of+dynamics+of+the+field+with+conference+data+in+DBLP&rft.jtitle=Scientometrics&rft.aulast=Song%2C+M.%3B+Heo%2C+G.E.%3B+Kim%2C+S.Y.&rft.au=Song%2C+M.%3B+Heo%2C+G.E.%3B+Kim%2C+S.Y.&rft.date=2014&rft.volume=101&rft.issue=1&rft.pages=397%E2%80%93428&rft_id=info:doi\/10.1007%2Fs11192-014-1246-2&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeeABird15-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeeABird15_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lee, D.; Kim, W.C.; Charidimou, A.; Song, M. (2015). \"A Bird's-Eye View of Alzheimer's Disease Research: Reflecting Different Perspectives of Indexers, Authors, or Citers in Mapping the Field\". <i>Journal of Alzheimer's Disease<\/i> <b>45<\/b> (4): 1207-22. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2FJAD-142688\" target=\"_blank\">10.3233\/JAD-142688<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25697702\" target=\"_blank\">25697702<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Bird%27s-Eye+View+of+Alzheimer%27s+Disease+Research%3A+Reflecting+Different+Perspectives+of+Indexers%2C+Authors%2C+or+Citers+in+Mapping+the+Field&rft.jtitle=Journal+of+Alzheimer%27s+Disease&rft.aulast=Lee%2C+D.%3B+Kim%2C+W.C.%3B+Charidimou%2C+A.%3B+Song%2C+M.&rft.au=Lee%2C+D.%3B+Kim%2C+W.C.%3B+Charidimou%2C+A.%3B+Song%2C+M.&rft.date=2015&rft.volume=45&rft.issue=4&rft.pages=1207-22&rft_id=info:doi\/10.3233%2FJAD-142688&rft_id=info:pmid\/25697702&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DingEntity13-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DingEntity13_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ding, Y.; Song, M.; Han, J. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3756961\" target=\"_blank\">\"Entitymetrics: Measuring the impact of entities\"<\/a>. <i>PLoS One<\/i> <b>8<\/b> (8): e71416. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0071416\" target=\"_blank\">10.1371\/journal.pone.0071416<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3756961\/\" target=\"_blank\">PMC3756961<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24009660\" target=\"_blank\">24009660<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3756961\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3756961<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Entitymetrics%3A+Measuring+the+impact+of+entities&rft.jtitle=PLoS+One&rft.aulast=Ding%2C+Y.%3B+Song%2C+M.%3B+Han%2C+J.+et+al.&rft.au=Ding%2C+Y.%3B+Song%2C+M.%3B+Han%2C+J.+et+al.&rft.date=2013&rft.volume=8&rft.issue=8&rft.pages=e71416&rft_id=info:doi\/10.1371%2Fjournal.pone.0071416&rft_id=info:pmc\/PMC3756961&rft_id=info:pmid\/24009660&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3756961&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HofmannProb99-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HofmannProb99_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hofmann, T. (1999). \"Probabilistic latent semantic indexing\". <i>Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval<\/i> <b>1999<\/b>: 50\u201357. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F312624.312649\" target=\"_blank\">10.1145\/312624.312649<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Probabilistic+latent+semantic+indexing&rft.jtitle=Proceedings+of+the+22nd+Annual+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Information+Retrieval&rft.aulast=Hofmann%2C+T.&rft.au=Hofmann%2C+T.&rft.date=1999&rft.volume=1999&rft.pages=50%E2%80%9357&rft_id=info:doi\/10.1145%2F312624.312649&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BleiLatent03-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BleiLatent03_25-0\" rel=\"external_link\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-BleiLatent03_25-1\" rel=\"external_link\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blei, D.M.; Ng, A.Y.; Jordan, M.I. (2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/jmlr.csail.mit.edu\/papers\/v3\/blei03a.html\" target=\"_blank\">\"Latent dirichlet allocation\"<\/a>. <i>Journal of Machine Learning Research<\/i> <b>3<\/b> (1): 993\u20131022<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/jmlr.csail.mit.edu\/papers\/v3\/blei03a.html\" target=\"_blank\">http:\/\/jmlr.csail.mit.edu\/papers\/v3\/blei03a.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Latent+dirichlet+allocation&rft.jtitle=Journal+of+Machine+Learning+Research&rft.aulast=Blei%2C+D.M.%3B+Ng%2C+A.Y.%3B+Jordan%2C+M.I.&rft.au=Blei%2C+D.M.%3B+Ng%2C+A.Y.%3B+Jordan%2C+M.I.&rft.date=2003&rft.volume=3&rft.issue=1&rft.pages=993%E2%80%931022&rft_id=http%3A%2F%2Fjmlr.csail.mit.edu%2Fpapers%2Fv3%2Fblei03a.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TangTopic11-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TangTopic11_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tang, J.; Zhang, J.; Jin, R. et al. (2011). \"Topic level expertise search over heterogeneous networks\". <i>Machine Learning<\/i> <b>82<\/b> (2): 211\u2013237. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10994-010-5212-9\" target=\"_blank\">10.1007\/s10994-010-5212-9<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Topic+level+expertise+search+over+heterogeneous+networks&rft.jtitle=Machine+Learning&rft.aulast=Tang%2C+J.%3B+Zhang%2C+J.%3B+Jin%2C+R.+et+al.&rft.au=Tang%2C+J.%3B+Zhang%2C+J.%3B+Jin%2C+R.+et+al.&rft.date=2011&rft.volume=82&rft.issue=2&rft.pages=211%E2%80%93237&rft_id=info:doi\/10.1007%2Fs10994-010-5212-9&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimExploring16-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KimExploring16_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, H.J.; An, J.; Jeong, Y.K.; Song, M. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dblp.uni-trier.de\/db\/conf\/jcdl\/birndl2016.html\" target=\"_blank\">\"Exploring the Leading Authors and Journals in Major Topics by Citation Sentences and Topic Modeling\"<\/a>. <i>Proceedings of the Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL) co-located with the Joint Conference on Digital Libraries 2016<\/i> <b>2016<\/b>: 42\u201350<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dblp.uni-trier.de\/db\/conf\/jcdl\/birndl2016.html\" target=\"_blank\">http:\/\/dblp.uni-trier.de\/db\/conf\/jcdl\/birndl2016.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+the+Leading+Authors+and+Journals+in+Major+Topics+by+Citation+Sentences+and+Topic+Modeling&rft.jtitle=Proceedings+of+the+Joint+Workshop+on+Bibliometric-enhanced+Information+Retrieval+and+Natural+Language+Processing+for+Digital+Libraries+%28BIRNDL%29+co-located+with+the+Joint+Conference+on+Digital+Libraries+2016&rft.aulast=Kim%2C+H.J.%3B+An%2C+J.%3B+Jeong%2C+Y.K.%3B+Song%2C+M.&rft.au=Kim%2C+H.J.%3B+An%2C+J.%3B+Jeong%2C+Y.K.%3B+Song%2C+M.&rft.date=2016&rft.volume=2016&rft.pages=42%E2%80%9350&rft_id=http%3A%2F%2Fdblp.uni-trier.de%2Fdb%2Fconf%2Fjcdl%2Fbirndl2016.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MedelyanHuman09-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MedelyanHuman09_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Medelyan, O. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/hdl.handle.net\/10289\/3513\" target=\"_blank\">\"Human-competitive automatic topic indexing (Thesis)\"<\/a>. The University of Waikato<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/hdl.handle.net\/10289\/3513\" target=\"_blank\">http:\/\/hdl.handle.net\/10289\/3513<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Human-competitive+automatic+topic+indexing+%28Thesis%29&rft.atitle=&rft.aulast=Medelyan%2C+O.&rft.au=Medelyan%2C+O.&rft.date=2009&rft.pub=The+University+of+Waikato&rft_id=http%3A%2F%2Fhdl.handle.net%2F10289%2F3513&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WittenKEA99-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WittenKEA99_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Witten, I.H.; Paynter, G.W.; Frank, E. et al. (1999). \"KEA: Practical automatic keyphrase extraction\". <i>Proceedings of the Fourth ACM Conference on Digital Libraries<\/i> <b>1999<\/b>: 254\u2013255. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F313238.313437\" target=\"_blank\">10.1145\/313238.313437<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=KEA%3A+Practical+automatic+keyphrase+extraction&rft.jtitle=Proceedings+of+the+Fourth+ACM+Conference+on+Digital+Libraries&rft.aulast=Witten%2C+I.H.%3B+Paynter%2C+G.W.%3B+Frank%2C+E.+et+al.&rft.au=Witten%2C+I.H.%3B+Paynter%2C+G.W.%3B+Frank%2C+E.+et+al.&rft.date=1999&rft.volume=1999&rft.pages=254%E2%80%93255&rft_id=info:doi\/10.1145%2F313238.313437&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Grammar and word used were updated to make the text easier to read.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181451\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.722 seconds\nReal time usage: 0.753 seconds\nPreprocessor visited node count: 23081\/1000000\nPreprocessor generated node count: 34651\/1000000\nPost\u2010expand include size: 166644\/2097152 bytes\nTemplate argument size: 56010\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 684.264 1 - -total\n 83.87% 573.926 1 - Template:Reflist\n 75.40% 515.932 28 - Template:Cite_journal\n 73.96% 506.064 29 - Template:Citation\/core\n 9.95% 68.095 1 - Template:Infobox_journal_article\n 9.36% 64.037 1 - Template:Infobox\n 7.46% 51.018 36 - Template:Citation\/identifier\n 5.53% 37.838 80 - Template:Infobox\/row\n 3.55% 24.323 29 - Template:Citation\/make_link\n 2.88% 19.689 76 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10142-0!*!0!!en!5!* and timestamp 20181214181450 and revision id 30746\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique\">https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","55740ce4aa95ae034b56b161256958e3_images":["https:\/\/www.limswiki.org\/images\/4\/4c\/Fig1_Heo_BMCBioinformatics2017_18.gif","https:\/\/www.limswiki.org\/images\/d\/d7\/Fig2_Heo_BMCBioinformatics2017_18.gif","https:\/\/www.limswiki.org\/images\/2\/2b\/Fig3_Heo_BMCBioinformatics2017_18.gif","https:\/\/www.limswiki.org\/images\/0\/00\/Fig4_Heo_BMCBioinformatics2017_18.gif","https:\/\/www.limswiki.org\/images\/c\/cd\/Fig5_Heo_BMCBioinformatics2017_18.gif"],"55740ce4aa95ae034b56b161256958e3_timestamp":1544811290,"e5903ddcc7734415af1d91fcd258da90_type":"article","e5903ddcc7734415af1d91fcd258da90_title":"The FAIR Guiding Principles for scientific data management and stewardship (Wilkinson et al. 2016)","e5903ddcc7734415af1d91fcd258da90_url":"https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship","e5903ddcc7734415af1d91fcd258da90_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:The FAIR Guiding Principles for scientific data management and stewardship\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nThe FAIR Guiding Principles for scientific data management and stewardshipJournal\n \nScientific DataAuthor(s)\n \n\n List of authors\n Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J.; Appleton, G.; Axton, M.; Baak, A.; Blomberg, N.; Boiten, J.W.; da Silva Santos, L.B.; Bourne, P.E.; Bouwman, J.; Brookes, A.J.; Clark, T.; Crosas, M.; Dillo, I.; Dumon, O.; Edmunds, S.; Evelo, C.T.; Finkers, R.; Gonzalez-Beltran, A.; Gray, A.J.; Groth, P.; Goble, C.; Grethe, J.S.; Heringa, J.; 't Hoen, P.A.; Hooft, R.; Kuhn, T.; Kok, R.; Kok, J.; Lusher, S.J.; Martone, M.E.; Mons, A.; Packer, A.L.; Persson, B.; Rocca-Serra, P.; Roos, M.; van Schaik, R.; Sansone, S.A.; Schultes, E.; Sengstag, T.; Slater, T.; Strawn, G.; Swertz, M.A.; Thompson, M.; van der Lei, J.; van Mulligen, E.; Velterop, J.; Waagmeester, A.; Wittenburg, P.; Wolstencroft, K.; Zhao, J.; Mons, B.\nAuthor affiliation(s)\n \n\n List of author affiliations\n Universidad Polit\u00e9cnica de Madrid, Stanford University, Nature Genetics, Euretos and Phortos Consultants, Wellcome Genome Campus, Lygature, Vrije Universiteit Amsterdam, National Institutes of Health, TNO, University of Leicester, Harvard Medical School, Harvard University, Data Archiving and Networked Services at The Hague, Beijing Genomics Institute, Maastricht University, Wageningen UR Plant Breeding, University of Oxford, Heriot-Watt University, University of Manchester, University of California San Diego, Dutch Techcenter for the Life Sciences, VU University Amsterdam, Leiden University, Netherlands eScience Center, National Center for Microscopy and Imaging Research, Phortos Consultants, UNIFESP Foundation, Uppsala University, Leiden University Medical Center, Bayer CropScience, University of Basel, Cray, University of Groningen, Erasmus MC - Rotterdam, Independent Open Access and Open Science Advocate, Micelio, Max Planck Compute and Data Facility, Dutch TechCenter for Life Sciences\nPrimary contact\n \nE-mail: Barend Mons (must log in)Year published\n \n2016Volume and issue\n \n3Page(s)\n \n160018DOI\n \n10.1038\/sdata.2016.18ISSN\n \n2052-4463Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.nature.com\/articles\/sdata201618Download\n \nhttps:\/\/www.nature.com\/articles\/sdata201618.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Comment \n\n2.1 Supporting discovery through good data management \n2.2 The significance of machines in data-rich research environments \n2.3 The FAIR Guiding Principles in detail \n2.4 The Principles precede implementation \n2.5 Examples of FAIRness, and the resulting value-added Dataverse \n2.6 FAIRness is a prerequisite for proper data management and data stewardship \n\n\n3 Acknowledgements \n4 Author information \n\n4.1 Competing interests \n\n\n5 References \n6 Notes \n\n\n\nAbstract \nThere is an urgent need to improve the infrastructure supporting the reuse of scholarly data. A diverse set of stakeholders \u2014 representing academia, industry, funding agencies, and scholarly publishers \u2014 have come together to design and jointly endorse a concise and measureable set of principles that we refer to as the FAIR Data Principles. The intent is that these may act as a guideline for those wishing to enhance the reusability of their data holdings. Distinct from peer initiatives that focus on the human scholar, the FAIR Principles put specific emphasis on enhancing the ability of machines to automatically find and use the data, in addition to supporting its reuse by individuals. This comment article represents the first formal publication of the FAIR Principles, and it includes the rationale behind them as well as some exemplar implementations in the community.\n\nComment \nSupporting discovery through good data management \nGood data management is not a goal in itself but rather the key conduit leading to knowledge discovery and innovation, and to subsequent data and knowledge integration and reuse by the community after the data publication process. Unfortunately, the existing digital ecosystem surrounding scholarly data publication prevents us from extracting maximum benefit from our research investments (e.g., Roche et al.[1]). Partially in response to this, science funders, publishers and governmental agencies are beginning to require data management and stewardship plans for data generated in publicly funded experiments. Beyond proper collection, annotation, and archival purposes, data stewardship includes the notion of \"long-term care\" of valuable digital assets, with the goal that they should be discovered and re-used for downstream investigations, either alone, or in combination with newly generated data. The outcomes from good data management and stewardship, therefore, are high-quality digital publications that facilitate and simplify this ongoing process of discovery, evaluation, and reuse in downstream studies. What constitutes \"good data management\" is, however, largely undefined, and is generally left as a decision for the data or repository owner. Therefore, bringing some clarity around the goals and desiderata of good data management and stewardship, and defining simple guideposts to inform those who publish and\/or preserve scholarly data, would be of great utility.\nThis article describes four foundational principles \u2014 findability, accessibility, interoperability, and reusability \u2014 that serve to guide data producers and publishers as they navigate around these obstacles, thereby helping to maximize the added value gained by contemporary, formal scholarly digital publishing. Importantly, it is our intent that the principles apply not only to \"data\" in the conventional sense, but also to the algorithms, tools, and workflows that led to that data. All scholarly digital research objects[2] \u2014 from data to analytical pipelines \u2014 benefit from application of these principles, since all components of the research process must be available to ensure transparency, reproducibility, and reusability.\nThere are numerous and diverse stakeholders who stand to benefit from overcoming these obstacles: researchers wanting to share, get credit, and reuse each other\u2019s data and interpretations; professional data publishers offering their services; software and tool-builders providing data analysis and processing services such as reusable workflows; funding agencies (private and public) increasingly concerned with long-term data stewardship; and a data science community mining, integrating, and analyzing new and existing data to advance discovery. To facilitate the reading of this manuscript by these diverse stakeholders, we provide definitions for common abbreviations in Box 1. Humans, however, are not the only critical stakeholders in the milieu of scientific data. Similar problems are encountered by the applications and computational agents that we task to undertake data retrieval and analysis on our behalf. These \"computational stakeholders\" are increasingly relevant, and the demand as much, or more, attention as their importance grows. One of the grand challenges of data-intensive science, therefore, is to improve knowledge discovery through assisting both humans and their computational agents in the discovery of, access to, and integration and analysis of task-appropriate scientific data and other scholarly digital objects.\n\n\n\n\n\n\n\nBox 1: Terms and abbreviations\n\n\nBD2K \u2014 Big Data 2 Knowledge, a trans-NIH initiative established to enable biomedical research as a digital research enterprise, to facilitate discovery and support new knowledge, and to maximise community engagement\nDOI \u2014 Digital Object Identifier, a code used to permanently and stably identify (usually digital) objects; DOIs provide a standard mechanism for retrieval of metadata about the object, and generally a means to access the data object itself.\nFAIR \u2014 Findable, Accessible, Interoperable, Reusable\nFORCE11 \u2014 The Future of Research Communications and e-Scholarship, a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing; initiated in 2011\nInteroperability \u2014 The ability of data or tools from non-cooperating resources to integrate or work together with minimal effort\nJDDCP \u2014 Joint Declaration of Data Citation Principles, acknowledging data as a first-class research output and supporting good research practices around data re-use; JDDCP proposes a set of guiding principles for citation of data within scholarly literature, another dataset, or any other research object.\nRDF \u2014 Resource Description Framework, a globally-accepted framework for data and knowledge representation that is intended to be read and interpreted by machines\n\n\n\n\nFor certain types of important digital objects, there are well-curated, deeply integrated, special-purpose repositories such as GenBank[3], worldwide Protein Data Bank (wwPDB)[4], and UniProt[5] in the life sciences; Space Physics Data Facility (SPDF; http:\/\/spdf.gsfc.nasa.gov\/) and Set of Identifications, Measurements and Bibliography for Astronomical Data (SIMBAD)[6] in the space sciences. \nThese foundational and critical core resources are continuously curating and capturing high-value reference datasets and fine-tuning them to enhance scholarly output, provide support for both human and mechanical users, and provide extensive tooling to access their content in rich, dynamic ways. However, not all datasets or even data types can be captured by, or submitted to, these repositories. Many important datasets emerging from traditional, low-throughput bench science don\u2019t fit in the data models of these special-purpose repositories, yet these datasets are no less important with respect to integrative research, reproducibility, and reuse in general. Apparently in response to this, we see the emergence of numerous general-purpose data repositories, at scales ranging from institutional (for example, a single university), to open globally-scoped repositories such as Dataverse[7], FigShare (http:\/\/figshare.com), Dryad[8], Mendeley Data (https:\/\/data.mendeley.com\/), Zenodo (http:\/\/zenodo.org\/), DataHub (http:\/\/datahub.io), DANS (http:\/\/www.dans.knaw.nl\/), and EUDAT.[9] Such repositories accept a wide range of data types in a wide variety of formats, generally do not attempt to integrate or harmonize the deposited data, and place few restrictions (or requirements) on the descriptors of the data deposition. The resulting data ecosystem, therefore, appears to be moving away from centralization and is becoming more diverse and less integrated, thereby exacerbating the discovery and re-usability problem for both human and computational stakeholders.\nA specific example of these obstacles could be imagined in the domain of gene regulation and expression analysis. Suppose a researcher has generated a dataset of differentially selected polyadenylation sites in a non-model pathogenic organism grown under a variety of environmental conditions that stimulate its pathogenic state. The researcher is interested in comparing the alternatively polyadenylated genes in this local dataset to other examples of alternative polyadenylation as well as the expression levels of these genes \u2014 both in this organism and related model organisms \u2014 during the infection process. Given that there is no special-purpose archive for differential polyadenylation data and no model organism database for this pathogen, where does the researcher begin?\nWe will consider the current approach to this problem from a variety of data discovery and integration perspectives. If the desired datasets existed, where might they have been published, and how would one begin to search for them, using what search tools? The desired search would need to filter based on specific species, tissues, types of data (Poly-A, microarray, NGS), conditions (infection), and genes; is that information (\"metadata\") captured by the repositories, and if so, what format is it in, is it searchable, and how? Once the data is discovered, can it be downloaded? In what format(s)? Can that format be easily integrated with private in-house data (the local dataset of alternative polyadenylation sites) as well as other data publications from third parties and with the community\u2019s core gene\/protein data repositories? Can this integration be done automatically to save time and avoid copy\/paste errors? Does the researcher have permission to use the data from these third-party researchers, under what license conditions, and who should be cited if a data-point is reused?\nQuestions such as these highlight some of the barriers to data discovery and reuse, not only for humans, but even more so for machines; yet it is precisely these kinds of deeply and broadly integrative analyses that constitute the bulk of contemporary e-Science. The reason that we often need several weeks (or months) of specialist technical effort to gather the data necessary to answer such research questions is not the lack of appropriate technology; the reason is, that we do not pay our valuable digital objects the careful attention they deserve when we create and preserve them. Overcoming these barriers, therefore, necessitates that all stakeholders \u2014 including researchers, special-purpose, and general-purpose repositories \u2014 evolve to meet the emergent challenges described above. The goal is for scholarly digital objects of all kinds to become \"first class citizens\" in the scientific publication ecosystem, where the quality of the publication \u2014 and more importantly, the impact of the publication \u2014 is a function of its ability to be accurately and appropriately found, re-used, and cited over time, by all stakeholders, both human and mechanical.\nWith this goal in-mind, a workshop was held in Leiden, Netherlands, in 2014, named Jointly Designing a Data Fairport. This workshop brought together a wide group of academic and private stakeholders all of whom had an interest in overcoming data discovery and reuse obstacles. From the deliberations at the workshop the notion emerged that through the definition of, and widespread support for, a minimal set of community-agreed guiding principles and practices, all stakeholders could more easily discover, access, appropriately integrate and re-use, and adequately cite, the vast quantities of information being generated by contemporary data-intensive science. The meeting concluded with a draft formulation of a set of foundational principles that were subsequently elaborated in greater detail: namely, that all research objects should be findable, accessible, interoperable and reusable (FAIR) both for machines and for people. These are now referred to as the FAIR Guiding Principles. Subsequently, a dedicated FAIR working group, established by several members of the FORCE11 community[10] fine-tuned and improved the Principles. The results of these efforts are reported here.\n\nThe significance of machines in data-rich research environments \nThe emphasis placed on FAIRness being applied to both human-driven and machine-driven activities, is a specific focus of the FAIR Guiding Principles that distinguishes them from many peer initiatives (discussed in the subsequent section). Humans and machines often face distinct barriers when attempting to find and process data on the web. Humans have an intuitive sense of \"semantics\" (the meaning or intent of a digital object) because we are capable of identifying and interpreting a wide variety of contextual cues, whether those take the form of structural\/visual\/iconic cues in the layout of a web page, or the content of narrative notes. As such, we are less likely to make errors in the selection of appropriate data or other digital objects, although humans will face similar difficulties if sufficient contextual metadata is lacking. The primary limitation of humans, however, is that we are unable to operate at the scope, scale, and speed necessitated by the scale of contemporary scientific data and complexity of e-Science. It is for this reason that humans increasingly rely on computational agents to undertake discovery and integration tasks on their behalf. This necessitates machines to be capable of autonomously and appropriately acting when faced with the wide range of types, formats, and access-mechanisms\/protocols that will be encountered during their self-guided exploration of the global data ecosystem. It also necessitates that the machines keep an exquisite record of provenance such that the data they are collecting can be accurately and adequately cited. Assisting these agents, therefore, is a critical consideration for all participants in the data management and stewardship process \u2014 from researchers and data producers to data repository hosts.\nThroughout this paper, we use the phrase \"machine actionable\" to indicate a continuum of possible states wherein a digital object provides increasingly more detailed information to an autonomously acting, computational data explorer. This information enables the agent \u2014 to a degree dependent on the amount of detail provided \u2014 to have the capacity, when faced with a digital object never encountered before, to: a) identify the type of object (with respect to both structure and intent); b) determine if it is useful within the context of the agent\u2019s current task by interrogating metadata and\/or data elements; c) determine if it is usable, with respect to license, consent, or other accessibility or use constraints; and d) take appropriate action, in much the same manner that a human would.\nFor example, a machine may be capable of determining the data-type of a discovered digital object, but not capable of parsing it due to it being in an unknown format; or it may be capable of processing the contained data, but not capable of determining the licensing requirements related to the retrieval and\/or use of that data. The optimal state \u2014 where machines fully \"understand\" and can autonomously and correctly operate, on a digital object \u2014 may rarely be achieved. Nevertheless, the FAIR principles provide \"steps along a path\" toward machine-actionability; adopting, in whole or in part, the FAIR principles, leads the resource along the continuum towards this optimal state. In addition, the idea of being machine-actionable applies in two contexts, first, when referring to the contextual metadata surrounding a digital object (\"what is it?\"), and second, when referring to the content of the digital object itself (\"how do I process it\/integrate it?\"). Either or both of these may be machine-actionable, and each forms its own continuum of actionability.\nFinally, we wish to draw a distinction between data that is machine-actionable as a result of specific investment in software supporting that data-type, for example, bespoke parsers that understand life science wwPDB files or space science Space Physics Archive Search and Extract (SPASE) files, and data that is machine-actionable exclusively through the utilization of general-purpose, open technologies. To reiterate the earlier point\u2014ultimate machine-actionability occurs when a machine can make a useful decision regarding data that it has not encountered before. This distinction is important when considering both (a) the rapidly growing and evolving data environment, with new technologies and new, more complex data-types continuously being developed, and (b) the growth of general-purpose repositories, where the data-types likely to be encountered by an agent are unpredictable. Creating bespoke parsers, in all computer languages for all data-types and all analytical tools that require those data-types, is not a sustainable activity. As such, the focus on assisting machines in their discovery and exploration of data through application of more generalized interoperability technologies and standards at the data\/repository level becomes a top-level priority for good data stewardship.\n\nThe FAIR Guiding Principles in detail \nRepresentatives of the interested stakeholder groups, discussed above, coalesced around four core desiderata \u2014 the FAIR Guiding Principles \u2014 and limited elaboration of these, which have been refined (Box 2) from the meeting\u2019s original draft, available at https:\/\/www.force11.org\/node\/6062. A separate document that dynamically addresses community discussion relating to clarifications and explanations of the principles, and detailed guidelines for and examples of FAIR implementations, is currently being constructed (http:\/\/datafairport.org\/fair-principles-living-document-menu). The FAIR Guiding Principles describe distinct considerations for contemporary data publishing environments with respect to supporting both manual and automated deposition, exploration, sharing, and reuse. While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival[1][11][12], FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs. Throughout the Principles, we use the phrase \"(meta)data\" in cases where the Principle should be applied to both metadata and data.\n\n\n\n\n\n\n\nBox 2: The FAIR Guiding Principles\n\n\nTo be findable:\nF1. (meta)data are assigned a globally unique and persistent identifier\nF2. data are described with rich metadata (defined by R1 below)\nF3. metadata clearly and explicitly include the identifier of the data it describes\nF4. (meta)data are registered or indexed in a searchable resource\nTo be accessible:\nA1. (meta)data are retrievable by their identifier using a standardized communications protocol\nA1.1 the protocol is open, free, and universally implementable\nA1.2 the protocol allows for an authentication and authorization procedure, where necessary\nA2. metadata are accessible, even when the data are no longer available\nTo be interoperable:\nI1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.\nI2. (meta)data use vocabularies that follow FAIR principles\nI3. (meta)data include qualified references to other (meta)data\nTo be reusable:\nR1. meta(data) are richly described with a plurality of accurate and relevant attributes\nR1.1. (meta)data are released with a clear and accessible data usage license\nR1.2. (meta)data are associated with detailed provenance\nR1.3. (meta)data meet domain-relevant community standards\n\n\n\n\nThe elements of the FAIR Principles are related, but independent and separable. The Principles define characteristics that contemporary data resources, tools, vocabularies and infrastructures should exhibit to assist discovery and reuse by third parties. By minimally defining each guiding principle, the barrier-to-entry for data producers, publishers, and stewards who wish to make their data holdings FAIR is purposely maintained as low as possible. The Principles may be adhered to in any combination and incrementally, as data providers\u2019 publishing environments evolve to increasing degrees of \"FAIRness.\" Moreover, the modularity of the Principles, and their distinction between data and metadata, explicitly support a wide range of special circumstances. One such example is highly sensitive or personally identifiable data, where publication of rich metadata to facilitate discovery, including clear rules regarding the process for accessing the data, provides a high degree of \"FAIRness\" even in the absence of FAIR publication of the data itself. A second example involves the publication of non-data research objects. Analytical workflows, for example, are a critical component of the scholarly ecosystem, and their formal publication is necessary to achieve both transparency and scientific reproducibility. The FAIR principles can equally be applied to these non-data assets, which need to be identified, described, discovered, and reused in much the same manner as data.\nSpecific exemplar efforts that provide varying levels of FAIRness are detailed later in this document. Additional issues, however, remain to be addressed. First, when community-endorsed vocabularies or other (meta)data standards do not include the attributes necessary to achieve rich annotation, there are two possible solutions: either publish an extension of an existing, closely related vocabulary, or \u2014 in the extreme case \u2014 create and explicitly publish a new vocabulary resource, following FAIR principles (\"I2\"). Second, for explicitly identifying the standard chosen when more than one vocabulary or other (meta)data standard is available, and given that for instance in the life sciences there are over 600 content standards, the BioSharing registry (https:\/\/biosharing.org\/) can be of use as it describes the standards in detail, including versions where applicable.\n\nThe Principles precede implementation \nThese high-level FAIR Guiding Principles precede implementation choices and do not suggest any specific technology, standard, or implementation-solution; moreover, the Principles are not, themselves, a standard or a specification. They act as a guide to data publishers and stewards to assist them in evaluating whether their particular implementation choices are rendering their digital research artifacts findable, accessible, interoperable, and reusable. We anticipate that these high-level principles will enable a broad range of integrative and exploratory behaviors, based on a wide range of technology choices and implementations. Indeed, many repositories are already implementing various aspects of FAIR using a variety of technology choices, and several examples are detailed in the next section; examples include scientific data itself and how narrative data articles are anchored to a progressively FAIR structured metadata.\n\nExamples of FAIRness, and the resulting value-added Dataverse \nDataverse[7] is an open-source data repository software installed in dozens of institutions globally to support public community repositories or institutional research data repositories. Harvard Dataverse, with more than 60,000 datasets, is the largest of the current Dataverse repositories and is open to all researchers from all research fields. Dataverse generates a formal citation for each deposit, following the standard defined by Altman and King.[13] Dataverse makes the Digital Object Identifier (DOI) or other persistent identifiers (Handles) public when the dataset is published (\"F\"). This resolves to a landing page, providing access to metadata, data files, dataset terms, waivers or licenses, and version information, all of which is indexed and searchable (\"F,\" \"A,\" and \"R\"). Deposits include metadata, data files, and any complementary files (such as documentation or code) needed to understand the data and analysis (\"R\"). Metadata is always public, even if the data are restricted or removed for privacy issues (\"F,\" \"A\"). This metadata is offered at three levels, extensively supporting the \"I\" and \"R\" FAIR principles: 1) data citation metadata, which maps to DataCite schema or Dublin Core Terms; 2) domain-specific metadata, which when possible maps to metadata standards used within a scientific domain; and 3) file-level metadata, which can be deep and extensive for tabular data files (including column-level metadata). Finally, Dataverse provides public machine-accessible interfaces to search the data, access the metadata, and download the data files, using a token to grant access when data files are restricted (\"A\").\nFAIRDOM (http:\/\/fair-dom.org\/about): integrates the SEEK[14] and openBIS[15] platforms to produce a FAIR data and model management facility for Systems Biology. Individual research assets (or aggregates of data and models) are identified with unique and persistent HTTP URLs, which can be registered with DOIs for publication (\"F\"). Assets can be accessed over the internet in a variety of formats appropriate for individuals and\/or their computers (RDF, XML) (\"I\"). Research assets are annotated with rich metadata, using community standards, formats and ontologies (\"I\"). The metadata is stored as RDF to enable interoperability and assets can be downloaded for reuse (\"R\").\nISA: is a community-driven metadata tracking framework to facilitate standards-compliant collection, curation, management and reuse of life science datasets.[16] ISA provides progressively FAIR structured metadata to Nature Scientific Data\u2019s Data Descriptor articles, and many GigaScience data papers, and underpins the EBI MetaboLights database among other data resources. At the core is a general-purpose, extensible ISA model, originally only available as a tabular representation but subsequently enhanced as an RDF-based representation[17], and JSON serializations to enable the \"I\" and \"R,\" becoming \"FAIR\" when published as linked data (http:\/\/elixir-uk.org\/node-events\/201cisa-as-a-fair-research-object201d-hack-the-spec-event-1) and complementing other research objects.[18]\nOpen PHACTS: Open PHACTS[19] is a data integration platform for information pertaining to drug discovery. Access to the platform is mediated through a machine-accessible interface[20] which provides multiple representations that are both human (HTML) and machine readable (RDF, JSON, XML, CSV, etc.), providing the \"A\" facet of FAIRness. The interface allows multiple URLs to be used to access information about a particular entity through a mapping service (\"F\" and \"A\"). Thus, a user can provide a ChEMBL URL to retrieve information sourced from, for example, Chemspider or DrugBank. Each call provides a canonical URL in its response (\"A\" and \"I\"). All data sources used are described using standardized dataset descriptions, following the global VoID standard, with rich provenance (\"R\" and \"I\"). All interface features are described using RDF following the Linked Data API specification (\"A\"). Finally, a majority of the datasets are described using community agreed upon ontologies (\"I\").\nwwPDB: wwPDB[4][21] is a special-purpose, intensively curated data archive that hosts information about experimentally determined 3D structures of proteins and nucleic acids. All wwPDB entries are stably hosted on an FTP server (\"A\") and represented in machine-readable formats (text and XML); the latter are machine-actionable using the metadata provided by the wwPDB conforming to the Macromolecular Information Framework (mmCIF)[22], a data standard of the International Union of Crystallography (IUCr) (\"F,\" \"I\" for humans and IUCr-aware machines). The wwPDB metadata contains cross-references to common identifiers such as PubMed and NCBI Taxonomy, and their wwPDB metadata are described in data dictionaries and schema documents (http:\/\/mmcif.wwpdb.org and http:\/\/pdbml.wwpdb.org) which conform to the IUCr data standard for the chemical and structural biology domains (\"R\"). A variety of software tools are available to interpret both wwPDB data and meta-data (\"I,\" \"R\" for humans and machines with this software). Each entry is represented by a DOI (\"F,\" \"A\" for humans and machines). The DOI resolves to a zipped file which requires special software for further interrogation\/interpretation. Other wwPDB access points[23][24][25] provide access to wwPDB records through URLs that are likely to be stable in the long-term (\"F\"), and all data and metadata is searchable through one or more of the wwPDB-affiliated websites (\"F\").\nUniProt: UniProt[5] is a comprehensive resource for protein sequence and annotation data. All entries are uniquely identified by a stable URL, that provides access to the record in a variety of formats, including a web page, plain-text, and RDF (\"F\" and \"A\"). The record contains rich metadata (\"F\") that is both human-readable (HTML) and machine-readable (text and RDF), where the RDF-formatted response utilizes shared vocabularies and ontologies such as UniProt Core, FALDO, and ECO (\"I\"). Interlinking with more than 150 different databases, every UniProt record has extensive links into, for example, PubMed, enabling rich citation. These links are machine-actionable in the RDF representation (\"R\"). Finally, in the RDF representation, the UniProt Core Ontology explicitly types all records, leaving no ambiguity \u2014 neither for humans nor machines \u2014 about what the data represents (\"R\"), enabling fully-automated retrieval of records and cross-referencing information.\nIn addition to, and in support of, communities and resources that are already pursuing FAIR objectives, the Data Citation Implementation Group of Force11 has published specific technical recommendations for how to implement many of the principles[26], with a particular focus on identifiers and their resolution, persistence, and metadata accessibility especially related to citation. In addition, the \"Skunkworks\" group that emerged from the Lorentz Workshop has been creating software supporting infrastructures[27] that are, end-to-end, compatible with FAIR principles, and can be implemented over existing repositories. These code modules have a particular focus on metadata publication and searchability, compatibility in cases of strict privacy considerations, and the extremely difficult problem of data and metadata interoperability (manuscript in preparation). Finally, there are several emergent projects, some listed in Box 3, for which FAIR is a key objective. These projects may provide valuable advice and guidance for those wishing to become more FAIR.\n\n\n\n\n\n\n\nBox 3: Emergent community\/collaborative initiatives with FAIR as a core focus or activity\n\n\nbioCADDIE (https:\/\/biocaddie.org): The NIH BD2K biomedical and healthCAre Data Discovery Index Ecosystem (bioCADDIE) consortium works to develop a Data Discovery Index (DDI) prototype, which is set to be as transformative and impactful for data as PubMed for biomedical literature.[28] The DDI focuses on finding (\"F\") and accessing (\"A\") the datasets stored across different sources, and it progressively works to identify relevant metadata[29] (\"I\") and maps them to community standards (\"R\"), linking to BioSharing.\nCEDAR: The Center for Expanded Data Annotation and Retrieval (CEDAR)[30] is an NIH BD2K funded center of excellence to develop tools and technologies that reduce the burden of authoring and enhancing metadata that meet community-based standards. CEDAR will enable the creation of metadata templates that implement community-based standards for experimental metadata, from BioSharing (https:\/\/biosharing.org), and that will be uniquely identifiable and retrievable with HTTP URIs, and annotated with vocabularies and ontologies drawn from BioPortal (http:\/\/bioportal.bioontology.org) (\"F,\" \"A,\" \"I,\" \"R\"). These templates will guide users to create rich metadata with unique and stable HTTP identifiers (\"F\") that can be retrieved using HTTP (\"A\") and accessible in a variety of formats (JSON-LD, TURTLE, RDF\/XML, CSV, etc.) (\"I\"). These metadata will use community standards, as defined by the template, and include provenance and data usage (\"R\").\nThese two projects, among others, provide tools and\/or collaborative opportunities for those who wish to improve the FAIRness of their data.\n\n\n\n\nFAIRness is a prerequisite for proper data management and data stewardship \nThe ideas within the FAIR Guiding Principles reflect, combine, build upon, and extend previous work by both the Concept Web Alliance (https:\/\/conceptweblog.wordpress.com\/) partners, who focused on machine-actionability and harmonization of data structures and semantics, and by the scientific and scholarly organizations that developed the Joint Declaration of Data Citation Principles (JDDCP)[31], who focused on primary scholarly data being made citable, discoverable, and available for reuse, so as to be capable of supporting more rigorous scholarship. An attempt to define the similarities and overlaps between the FAIR Principles and the JDDCP is provided at https:\/\/www.force11.org\/node\/6062. The FAIR Principles are also complementary to the Data Seal of Approval (DSA) (http:\/\/datasealofapproval.org\/media\/filer_public\/2013\/09\/27\/guidelines_2014-2015.pdf) in that they share the general aim to render data re-usable for users other than those who originally generated them. While the DSA focuses primarily on the responsibilities and conduct of data producers and repositories, FAIR focuses primarily on the data itself. Clearly, the broader community of stakeholders is coalescing around a set of common, dovetailed visions spanning all facets of the scholarly data publishing ecosystem.\nThe end result, when implemented, will be more rigorous management and stewardship of these valuable digital resources, to the benefit of the entire academic community. As stated at the outset, good data management and stewardship is not a goal in itself, but rather a precondition supporting knowledge discovery and innovation. Contemporary e-Science requires data to be findable, accessible, interoperable, and reusable in the long-term, and these objectives are rapidly becoming expectations of agencies and publishers. We demonstrate, therefore, that the FAIR Data Principles provide a set of mileposts for data producers and publishers. They guide the implementation of the most basic levels of good Data Management and Stewardship practice, thus helping researchers adhere to the expectations and requirements of their funding agencies. We call on all data producers and publishers to examine and implement these principles, and actively participate with the FAIR initiative by joining the Force11 working group. By working together towards shared, common goals, the valuable data produced by our community will gradually achieve the critical goals of FAIRness.\n\nAcknowledgements \nThe original Lorentz Workshop \"Jointly Designing a Data FAIRport\" was organized by Barend Mons in collaboration with and co-sponsored by the Lorentz center, The Dutch Techcenter for the Life Sciences, and the Netherlands eScience Center. The principles and themes described in this manuscript represent the significant voluntary contributions and participation of the authors at, and\/or subsequent to, this workshop and from the wider FORCE11, BD2K, and ELIXIR communities. We also acknowledge and thank the organizers and backers of the NBDC\/DBCLS BioHackathon 2015, where several of the authors made significant revisions to the FAIR Principles.\n\nAuthor information \nM.W. was the primary author of the manuscript and participated extensively in the drafting and editing of the FAIR Principles. M.D. was significantly involved in the drafting of the FAIR Principles. B.M. conceived of the FAIR Data Initiative and contributed extensively to the drafting of the principles and this manuscript text. All other authors are listed alphabetically, and contributed to the manuscript either by their participation in the initial workshop and\/or by editing or commenting on the manuscript text.\n\nCompeting interests \nM.A. is Nature Genetics\u2019 Editor in Chief; S.A.S. is Scientific Data\u2019s Honorary Academic Editor and consultant.\n\nReferences \n\n\n\u2191 1.0 1.1 Roche, D.G.; Kruuk, L.E.; Lanfear, R.; Binning, S.A. (2015). \"Public data archiving in ecology and evolution: How well are we doing?\". PLOS Biology 13: e1002295. doi:10.1371\/journal.pbio.1002295. PMC PMC4640582. PMID 26556502. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4640582 .   \n\n\u2191 Bechhofer, S.; De Roure, D.; Gamble, M. et al. (2010). \"Research objects: Towards exchange and reuse of digital knowledge\". Nature Precedings. doi:10.1038\/npre.2010.4626.1.   \n\n\u2191 Benson, D.A.; Cavanaugh, M.; Clark, K. et al. (2013). \"GenBank\". Nucleic Acids Research 41 (D1): D36-42. doi:10.1093\/nar\/gks1195. PMC PMC4640582. PMID PMC3531190. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4640582 .   \n\n\u2191 4.0 4.1 Berman, H.; Henrick, K.; Nakamura, H. (2003). \"Announcing the worldwide Protein Data Bank\". Nature Structural Biology 10 (12): 980. doi:10.1038\/nsb1203-980. PMID 14634627.   \n\n\u2191 5.0 5.1 UniProt Consortium (2015). \"UniProt: A hub for protein information\". Nucleic Acids Research 43 (D1): D204-12. doi:10.1093\/nar\/gku989. PMC PMC4384041. PMID 25348405. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4384041 .   \n\n\u2191 Wenger, M.; Ochsenbein, F.; Egret, D. et al. (2000). \"The SIMBAD astronomical database: The CDS reference database for astronomical objects\". Astronomy and Astrophysics Supplement Series 143 (1): 9\u201322. doi:10.1051\/aas:2000332.   \n\n\u2191 7.0 7.1 Crosas, M. (2011). \"The Dataverse Network: An open-source application for sharing, discovering and preserving data\". D-Lib Magazine 17 (1\/2): 2. doi:10.1045\/january2011-crosas.   \n\n\u2191 White, H.C.; Carrier, S.; Thompson, A. et al. (2008). \"The Dryad Data Repository: A Singapore Framework metadata architecture in a DSpace environment\". DC-2008--Berlin Proceedings 2008: 157\u2013162. http:\/\/dcpapers.dublincore.org\/pubs\/article\/view\/928 .   \n\n\u2191 Lecarpentier, D.; Wittenburg, P.; Elbers, W. (2013). \"EUDAT: A new cross-disciplinary data infrastructure for science\". International Journal of Digital Curation 8 (1): 279\u2013287. doi:10.2218\/ijdc.v8i1.260.   \n\n\u2191 Martone, M.E. (2015). \"FORCE11: Building the Future for Research Communications and e-Scholarship\". BioScience 65 (7): 635. doi:10.1093\/biosci\/biv095.   \n\n\u2191 White, E.P.; Baldridge, E.; Brym, Z.T. et al. (2013). \"Nine simple ways to make it easier to (re)use your data\". Ideas in Ecology and Evolution 6 (2). http:\/\/ojs.library.queensu.ca\/index.php\/IEE\/article\/view\/4608\/0 .   \n\n\u2191 Sandve, G.K.; Nekrutenko, A.; Taylor, J.; Hovig, E. (2013). \"Ten simple rules for reproducible computational research\". PLOS Computational Biology 9 (10). doi:10.1371\/journal.pcbi.1003285. PMC PMC3812051. PMID 24204232. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3812051 .   \n\n\u2191 Altman, M.; King, G. (2007). \"A proposed standard for the scholarly citation of quantitative data\". D-Lib Magazine 13 (3\/4). doi:10.1045\/march2007-altman.   \n\n\u2191 Wolstencroft, K.; Owen, S.; Krebs, O. et al. (2015). \"SEEK: A systems biology data and model management platform\". BMC Systems Biology 9: 33. doi:10.1186\/s12918-015-0174-y. PMC PMC4702362. PMID 26160520. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4702362 .   \n\n\u2191 Bauch, A.; Adamczyk, I.; Buczek, P. et al. (2011). \"openBIS: A flexible framework for managing and analyzing complex data in biology research\". BMC Bioinformatics 12: 468. doi:10.1186\/1471-2105-12-468. PMC PMC3275639. PMID 22151573. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639 .   \n\n\u2191 Sansone, S.A.; Rocca-Serra, P.; Field, D. et al. (2012). \"Toward interoperable bioscience data\". Nature Genetics 44 (2): 121\u20136. doi:10.1038\/ng.1054. PMC PMC3428019. PMID 22281772. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3428019 .   \n\n\u2191 Lam, K.K.; Khalak, A.; Tse, D. (2014). \"Near-optimal assembly for shotgun sequencing with noisy reads\". BMC Bioinformatics 15 (Supp. 9): S4. doi:10.1186\/1471-2105-15-S9-S4. PMC PMC4168708. PMID 25252708. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4168708 .   \n\n\u2191 Gonz\u00e1lez-Beltr\u00e1n, A.; Li, P.; Zhao, J. et al. (2015). \"From peer-reviewed to peer-reproduced in scholarly publishing: The complementary roles of data models and workflows in bioinformatics\". PLOS One 10 (7): e0127612. doi:10.1371\/journal.pone.0127612. PMC PMC4495984. PMID 26154165. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4495984 .   \n\n\u2191 Harland, L. (2012). \"Open PHACTS: A semantic knowledge infrastructure for public and commercial drug discovery research\". Knowledge Engineering and Knowledge Management - EKAW 2012 7603: 1\u20137. doi:10.1007\/978-3-642-33876-2_1.   \n\n\u2191 Groth, P.; Loizou, A.; Gray, A.J.G. et al. (2014). \"API-centric Linked Data integration: The Open PHACTS Discovery Platform case study\". Web Semantics: Science, Services and Agents on the World Wide Web 29: 12\u201318. doi:10.1016\/j.websem.2014.03.003.   \n\n\u2191 Berman, H.M.; Westbrook, J.; Feng, Z. et al. (2000). \"The Protein Data Bank\". Nucleic Acids Research 28 (1): 235-42. PMC PMC102472. PMID 10592235. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC102472 .   \n\n\u2191 Bourne, P.E.; Berman, H.M.; McMahon, B. et al. (1997). \"Macromolecular Crystallographic Information File\". Methods in Enzymology 277: 571\u201390. PMID 18488325.   \n\n\u2191 Rose, P.W.; Prli\u0107, A.; Bi, C. et al. (2015). \"The RCSB Protein Data Bank: Views of structural biology for basic and applied research and education\". Nucleic Acids Research 43 (D1): D345-56. doi:10.1093\/nar\/gku1214. PMC PMC4383988. PMID 25428375. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383988 .   \n\n\u2191 Kinjo, A.R.; Suzuki, H.; Yamashita, R. et al. (2012). \"Protein Data Bank Japan (PDBj): Maintaining a structural data archive and resource description framework format\". Nucleic Acids Research 40 (D1): D453-60. doi:10.1093\/nar\/gkr811. PMC PMC3245181. PMID 21976737. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3245181 .   \n\n\u2191 Gutmanas, A.; Alhroub, Y.; Battle, G.M. et al. (2014). \"PDBe: Protein Data Bank in Europe\". Nucleic Acids Research 42 (D1): D285-91. doi:10.1093\/nar\/gkt1180. PMC PMC3965016. PMID 24288376. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3965016 .   \n\n\u2191 Starr, J.; Castro, E.; Crosas, M. et al. (2015). \"Achieving human and machine accessibility of cited data in scholarly publications\". PeerJ Computer Science 1: e1. doi:10.7717\/peerj-cs.1. PMC PMC4498574. PMID 26167542. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4498574 .   \n\n\u2191 Wilkinson, M.; Dumontier, M.; Durbin, P. (10 November 2015). \"DataFairPort: The Perl libraries version 0.231\". Zenodo. doi:http:\/\/doi.org\/10.5281\/zenodo.33584.   \n\n\u2191 Ohno-Machado, L.; Alter, G.; Fore, I. et al. (08 March 2015). \"bioCADDIE White Paper: Vision and Implementation\". figshare. doi:10.6084\/m9.figshare.1362572. https:\/\/figshare.com\/articles\/bioCADDIE_white_paper_Data_Discovery_Index\/1362572 .   \n\n\u2191 WG3 Members (14 August 2015). \"WG3-MetadataSpecifications: NIH BD2K bioCADDIE Data Discovery Index WG3 Metadata Specification v1\". Zenodo. doi:10.5281\/zenodo.28019.   \n\n\u2191 Musen, M.A.; Bean, C.A.; Cheung, K.H. et al. (2015). \"The center for expanded data annotation and retrieval\". JAMIA 22 (6): 1148-52. doi:10.1093\/jamia\/ocv048. PMC PMC5009916. PMID 26112029. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5009916 .   \n\n\u2191 Data Citation Synthesis Group (2014). \"Joint Declaration of Data Citation Principles\". In Martone, M.. FORCE11. https:\/\/www.force11.org\/datacitation .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. In the original, reference 26 accidentally duplicates reference five; 26 was omitted and five used in its place. The original put references 30\u201332 before reference 29; because this wiki puts references in order of appearance, order will slightly differ from the original.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\">https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on big dataLIMSwiki journal articles on informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 2 May 2017, at 21:28.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,067 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","e5903ddcc7734415af1d91fcd258da90_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:The FAIR Guiding Principles for scientific data management and stewardship<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>There is an urgent need to improve the infrastructure supporting the reuse of scholarly data. A diverse set of stakeholders \u2014 representing academia, industry, funding agencies, and scholarly publishers \u2014 have come together to design and jointly endorse a concise and measureable set of principles that we refer to as the FAIR Data Principles. The intent is that these may act as a guideline for those wishing to enhance the reusability of their data holdings. Distinct from peer initiatives that focus on the human scholar, the FAIR Principles put specific emphasis on enhancing the ability of machines to automatically find and use the data, in addition to supporting its reuse by individuals. This comment article represents the first formal publication of the FAIR Principles, and it includes the rationale behind them as well as some exemplar implementations in the community.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Comment\">Comment<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Supporting_discovery_through_good_data_management\">Supporting discovery through good data management<\/span><\/h3>\n<p>Good data management is not a goal in itself but rather the key conduit leading to knowledge discovery and innovation, and to subsequent data and knowledge integration and reuse by the community after the data publication process. Unfortunately, the existing digital ecosystem surrounding scholarly data publication prevents us from extracting maximum benefit from our research investments (e.g., Roche <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-RochePublic15_1-0\" class=\"reference\"><a href=\"#cite_note-RochePublic15-1\" rel=\"external_link\">[1]<\/a><\/sup>). Partially in response to this, science funders, publishers and governmental agencies are beginning to require data management and stewardship plans for data generated in publicly funded experiments. Beyond proper collection, annotation, and archival purposes, data stewardship includes the notion of \"long-term care\" of valuable digital assets, with the goal that they should be discovered and re-used for downstream investigations, either alone, or in combination with newly generated data. The outcomes from good data management and stewardship, therefore, are high-quality digital publications that facilitate and simplify this ongoing process of discovery, evaluation, and reuse in downstream studies. What constitutes \"good data management\" is, however, largely undefined, and is generally left as a decision for the data or repository owner. Therefore, bringing some clarity around the goals and desiderata of good data management and stewardship, and defining simple guideposts to inform those who publish and\/or preserve scholarly data, would be of great utility.\n<\/p><p>This article describes four foundational principles \u2014 findability, accessibility, interoperability, and reusability \u2014 that serve to guide data producers and publishers as they navigate around these obstacles, thereby helping to maximize the added value gained by contemporary, formal scholarly digital publishing. Importantly, it is our intent that the principles apply not only to \"data\" in the conventional sense, but also to the algorithms, tools, and workflows that led to that data. All scholarly digital research objects<sup id=\"rdp-ebb-cite_ref-BechhoferResearch10_2-0\" class=\"reference\"><a href=\"#cite_note-BechhoferResearch10-2\" rel=\"external_link\">[2]<\/a><\/sup> \u2014 from data to analytical pipelines \u2014 benefit from application of these principles, since all components of the research process must be available to ensure transparency, reproducibility, and reusability.\n<\/p><p>There are numerous and diverse stakeholders who stand to benefit from overcoming these obstacles: researchers wanting to share, get credit, and reuse each other\u2019s data and interpretations; professional data publishers offering their services; software and tool-builders providing data analysis and processing services such as reusable workflows; funding agencies (private and public) increasingly concerned with long-term data stewardship; and a data science community mining, integrating, and analyzing new and existing data to advance discovery. To facilitate the reading of this manuscript by these diverse stakeholders, we provide definitions for common abbreviations in Box 1. Humans, however, are not the only critical stakeholders in the milieu of scientific data. Similar problems are encountered by the applications and computational agents that we task to undertake data retrieval and analysis on our behalf. These \"computational stakeholders\" are increasingly relevant, and the demand as much, or more, attention as their importance grows. One of the grand challenges of data-intensive science, therefore, is to improve knowledge discovery through assisting both humans and their computational agents in the discovery of, access to, and integration and analysis of task-appropriate scientific data and other scholarly digital objects.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"1\"><b>Box 1: Terms and abbreviations<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>BD2K<\/b> \u2014 Big Data 2 Knowledge, a trans-NIH initiative established to enable biomedical research as a digital research enterprise, to facilitate discovery and support new knowledge, and to maximise community engagement\n<p><b>DOI<\/b> \u2014 Digital Object Identifier, a code used to permanently and stably identify (usually digital) objects; DOIs provide a standard mechanism for retrieval of metadata about the object, and generally a means to access the data object itself.\n<\/p><p><b>FAIR<\/b> \u2014 Findable, Accessible, Interoperable, Reusable\n<\/p><p><b>FORCE11<\/b> \u2014 The Future of Research Communications and e-Scholarship, a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing; initiated in 2011\n<\/p><p><b>Interoperability<\/b> \u2014 The ability of data or tools from non-cooperating resources to integrate or work together with minimal effort\n<\/p><p><b>JDDCP<\/b> \u2014 Joint Declaration of Data Citation Principles, acknowledging data as a first-class research output and supporting good research practices around data re-use; JDDCP proposes a set of guiding principles for citation of data within scholarly literature, another dataset, or any other research object.\n<\/p><p><b>RDF<\/b> \u2014 Resource Description Framework, a globally-accepted framework for data and knowledge representation that is intended to be read and interpreted by machines\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For certain types of important digital objects, there are well-curated, deeply integrated, special-purpose repositories such as GenBank<sup id=\"rdp-ebb-cite_ref-BensonGenBank13_3-0\" class=\"reference\"><a href=\"#cite_note-BensonGenBank13-3\" rel=\"external_link\">[3]<\/a><\/sup>, worldwide Protein Data Bank (wwPDB)<sup id=\"rdp-ebb-cite_ref-BermanAnnoun03_4-0\" class=\"reference\"><a href=\"#cite_note-BermanAnnoun03-4\" rel=\"external_link\">[4]<\/a><\/sup>, and UniProt<sup id=\"rdp-ebb-cite_ref-UniProtUniProt15_5-0\" class=\"reference\"><a href=\"#cite_note-UniProtUniProt15-5\" rel=\"external_link\">[5]<\/a><\/sup> in the life sciences; Space Physics Data Facility (SPDF; <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/spdf.gsfc.nasa.gov\/\" target=\"_blank\">http:\/\/spdf.gsfc.nasa.gov\/<\/a>) and Set of Identifications, Measurements and Bibliography for Astronomical Data (SIMBAD)<sup id=\"rdp-ebb-cite_ref-WengerTheSIMBAD00_6-0\" class=\"reference\"><a href=\"#cite_note-WengerTheSIMBAD00-6\" rel=\"external_link\">[6]<\/a><\/sup> in the space sciences. \n<\/p><p>These foundational and critical core resources are continuously curating and capturing high-value reference datasets and fine-tuning them to enhance scholarly output, provide support for both human and mechanical users, and provide extensive tooling to access their content in rich, dynamic ways. However, not all datasets or even data types can be captured by, or submitted to, these repositories. Many important datasets emerging from traditional, low-throughput bench science don\u2019t fit in the data models of these special-purpose repositories, yet these datasets are no less important with respect to integrative research, reproducibility, and reuse in general. Apparently in response to this, we see the emergence of numerous general-purpose data repositories, at scales ranging from institutional (for example, a single university), to open globally-scoped repositories such as Dataverse<sup id=\"rdp-ebb-cite_ref-CrosasTheDataverse11_7-0\" class=\"reference\"><a href=\"#cite_note-CrosasTheDataverse11-7\" rel=\"external_link\">[7]<\/a><\/sup>, FigShare (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/figshare.com\" target=\"_blank\">http:\/\/figshare.com<\/a>), Dryad<sup id=\"rdp-ebb-cite_ref-WhiteTheDryad08_8-0\" class=\"reference\"><a href=\"#cite_note-WhiteTheDryad08-8\" rel=\"external_link\">[8]<\/a><\/sup>, Mendeley Data (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/data.mendeley.com\/\" target=\"_blank\">https:\/\/data.mendeley.com\/<\/a>), Zenodo (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/zenodo.org\/\" target=\"_blank\">http:\/\/zenodo.org\/<\/a>), DataHub (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/datahub.io\" target=\"_blank\">http:\/\/datahub.io<\/a>), DANS (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.dans.knaw.nl\/\" target=\"_blank\">http:\/\/www.dans.knaw.nl\/<\/a>), and EUDAT.<sup id=\"rdp-ebb-cite_ref-LecarpentierEUDAT13_9-0\" class=\"reference\"><a href=\"#cite_note-LecarpentierEUDAT13-9\" rel=\"external_link\">[9]<\/a><\/sup> Such repositories accept a wide range of data types in a wide variety of formats, generally do not attempt to integrate or harmonize the deposited data, and place few restrictions (or requirements) on the descriptors of the data deposition. The resulting data ecosystem, therefore, appears to be moving away from centralization and is becoming more diverse and less integrated, thereby exacerbating the discovery and re-usability problem for both human and computational stakeholders.\n<\/p><p>A specific example of these obstacles could be imagined in the domain of gene regulation and expression analysis. Suppose a researcher has generated a dataset of differentially selected polyadenylation sites in a non-model pathogenic organism grown under a variety of environmental conditions that stimulate its pathogenic state. The researcher is interested in comparing the alternatively polyadenylated genes in this local dataset to other examples of alternative polyadenylation as well as the expression levels of these genes \u2014 both in this organism and related model organisms \u2014 during the infection process. Given that there is no special-purpose archive for differential polyadenylation data and no model organism database for this pathogen, where does the researcher begin?\n<\/p><p>We will consider the current approach to this problem from a variety of data discovery and integration perspectives. If the desired datasets existed, where might they have been published, and how would one begin to search for them, using what search tools? The desired search would need to filter based on specific species, tissues, types of data (Poly-A, microarray, NGS), conditions (infection), and genes; is that information (\"metadata\") captured by the repositories, and if so, what format is it in, is it searchable, and how? Once the data is discovered, can it be downloaded? In what format(s)? Can that format be easily integrated with private in-house data (the local dataset of alternative polyadenylation sites) as well as other data publications from third parties and with the community\u2019s core gene\/protein data repositories? Can this integration be done automatically to save time and avoid copy\/paste errors? Does the researcher have permission to use the data from these third-party researchers, under what license conditions, and who should be cited if a data-point is reused?\n<\/p><p>Questions such as these highlight some of the barriers to data discovery and reuse, not only for humans, but even more so for machines; yet it is precisely these kinds of deeply and broadly integrative analyses that constitute the bulk of contemporary e-Science. The reason that we often need several weeks (or months) of specialist technical effort to gather the data necessary to answer such research questions is not the lack of appropriate technology; the reason is, that we do not pay our valuable digital objects the careful attention they deserve when we create and preserve them. Overcoming these barriers, therefore, necessitates that all stakeholders \u2014 including researchers, special-purpose, and general-purpose repositories \u2014 evolve to meet the emergent challenges described above. The goal is for scholarly digital objects of all kinds to become \"first class citizens\" in the scientific publication ecosystem, where the quality of the publication \u2014 and more importantly, the impact of the publication \u2014 is a function of its ability to be accurately and appropriately found, re-used, and cited over time, by all stakeholders, both human and mechanical.\n<\/p><p>With this goal in-mind, a workshop was held in Leiden, Netherlands, in 2014, named <i>Jointly Designing a Data Fairport<\/i>. This workshop brought together a wide group of academic and private stakeholders all of whom had an interest in overcoming data discovery and reuse obstacles. From the deliberations at the workshop the notion emerged that through the definition of, and widespread support for, a minimal set of community-agreed guiding principles and practices, all stakeholders could more easily discover, access, appropriately integrate and re-use, and adequately cite, the vast quantities of information being generated by contemporary data-intensive science. The meeting concluded with a draft formulation of a set of foundational principles that were subsequently elaborated in greater detail: namely, that all research objects should be findable, accessible, interoperable and reusable (FAIR) both for machines and for people. These are now referred to as the FAIR Guiding Principles. Subsequently, a dedicated FAIR working group, established by several members of the FORCE11 community<sup id=\"rdp-ebb-cite_ref-MartoneForce11_15_10-0\" class=\"reference\"><a href=\"#cite_note-MartoneForce11_15-10\" rel=\"external_link\">[10]<\/a><\/sup> fine-tuned and improved the Principles. The results of these efforts are reported here.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_significance_of_machines_in_data-rich_research_environments\">The significance of machines in data-rich research environments<\/span><\/h3>\n<p>The emphasis placed on FAIRness being applied to both human-driven and machine-driven activities, is a specific focus of the FAIR Guiding Principles that distinguishes them from many peer initiatives (discussed in the subsequent section). Humans and machines often face distinct barriers when attempting to find and process data on the web. Humans have an intuitive sense of \"semantics\" (the meaning or intent of a digital object) because we are capable of identifying and interpreting a wide variety of contextual cues, whether those take the form of structural\/visual\/iconic cues in the layout of a web page, or the content of narrative notes. As such, we are less likely to make errors in the selection of appropriate data or other digital objects, although humans will face similar difficulties if sufficient contextual metadata is lacking. The primary limitation of humans, however, is that we are unable to operate at the scope, scale, and speed necessitated by the scale of contemporary scientific data and complexity of e-Science. It is for this reason that humans increasingly rely on computational agents to undertake discovery and integration tasks on their behalf. This necessitates machines to be capable of autonomously and appropriately acting when faced with the wide range of types, formats, and access-mechanisms\/protocols that will be encountered during their self-guided exploration of the global data ecosystem. It also necessitates that the machines keep an exquisite record of provenance such that the data they are collecting can be accurately and adequately cited. Assisting these agents, therefore, is a critical consideration for all participants in the data management and stewardship process \u2014 from researchers and data producers to data repository hosts.\n<\/p><p>Throughout this paper, we use the phrase \"machine actionable\" to indicate a continuum of possible states wherein a digital object provides increasingly more detailed information to an autonomously acting, computational data explorer. This information enables the agent \u2014 to a degree dependent on the amount of detail provided \u2014 to have the capacity, when faced with a digital object never encountered before, to: a) identify the type of object (with respect to both structure and intent); b) determine if it is useful within the context of the agent\u2019s current task by interrogating metadata and\/or data elements; c) determine if it is usable, with respect to license, consent, or other accessibility or use constraints; and d) take appropriate action, in much the same manner that a human would.\n<\/p><p>For example, a machine may be capable of determining the data-type of a discovered digital object, but not capable of parsing it due to it being in an unknown format; or it may be capable of processing the contained data, but not capable of determining the licensing requirements related to the retrieval and\/or use of that data. The optimal state \u2014 where machines fully \"understand\" and can autonomously and correctly operate, on a digital object \u2014 may rarely be achieved. Nevertheless, the FAIR principles provide \"steps along a path\" toward machine-actionability; adopting, in whole or in part, the FAIR principles, leads the resource along the continuum towards this optimal state. In addition, the idea of being machine-actionable applies in two contexts, first, when referring to the contextual metadata surrounding a digital object (\"what is it?\"), and second, when referring to the content of the digital object itself (\"how do I process it\/integrate it?\"). Either or both of these may be machine-actionable, and each forms its own continuum of actionability.\n<\/p><p>Finally, we wish to draw a distinction between data that is machine-actionable as a result of specific investment in software supporting that data-type, for example, bespoke parsers that understand life science wwPDB files or space science Space Physics Archive Search and Extract (SPASE) files, and data that is machine-actionable exclusively through the utilization of general-purpose, open technologies. To reiterate the earlier point\u2014ultimate machine-actionability occurs when a machine can make a useful decision regarding data that it has not encountered before. This distinction is important when considering both (a) the rapidly growing and evolving data environment, with new technologies and new, more complex data-types continuously being developed, and (b) the growth of general-purpose repositories, where the data-types likely to be encountered by an agent are unpredictable. Creating bespoke parsers, in all computer languages for all data-types and all analytical tools that require those data-types, is not a sustainable activity. As such, the focus on assisting machines in their discovery and exploration of data through application of more generalized interoperability technologies and standards at the data\/repository level becomes a top-level priority for good data stewardship.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_FAIR_Guiding_Principles_in_detail\">The FAIR Guiding Principles in detail<\/span><\/h3>\n<p>Representatives of the interested stakeholder groups, discussed above, coalesced around four core desiderata \u2014 the FAIR Guiding Principles \u2014 and limited elaboration of these, which have been refined (Box 2) from the meeting\u2019s original draft, available at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.force11.org\/node\/6062\" target=\"_blank\">https:\/\/www.force11.org\/node\/6062<\/a>. A separate document that dynamically addresses community discussion relating to clarifications and explanations of the principles, and detailed guidelines for and examples of FAIR implementations, is currently being constructed (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/datafairport.org\/fair-principles-living-document-menu\" target=\"_blank\">http:\/\/datafairport.org\/fair-principles-living-document-menu<\/a>). The FAIR Guiding Principles describe distinct considerations for contemporary data publishing environments with respect to supporting both manual and automated deposition, exploration, sharing, and reuse. While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival<sup id=\"rdp-ebb-cite_ref-RochePublic15_1-1\" class=\"reference\"><a href=\"#cite_note-RochePublic15-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WhiteNine13_11-0\" class=\"reference\"><a href=\"#cite_note-WhiteNine13-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SandveTen13_12-0\" class=\"reference\"><a href=\"#cite_note-SandveTen13-12\" rel=\"external_link\">[12]<\/a><\/sup>, FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs. Throughout the Principles, we use the phrase \"(meta)data\" in cases where the Principle should be applied to both metadata and data.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"1\"><b>Box 2: The FAIR Guiding Principles<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>To be findable<\/b>:\n<p>F1. (meta)data are assigned a globally unique and persistent identifier\n<\/p><p>F2. data are described with rich metadata (defined by R1 below)\n<\/p><p>F3. metadata clearly and explicitly include the identifier of the data it describes\n<\/p><p>F4. (meta)data are registered or indexed in a searchable resource\n<\/p><p><b>To be accessible:<\/b>\n<\/p><p>A1. (meta)data are retrievable by their identifier using a standardized communications protocol\n<\/p><p>A1.1 the protocol is open, free, and universally implementable\n<\/p><p>A1.2 the protocol allows for an authentication and authorization procedure, where necessary\n<\/p><p>A2. metadata are accessible, even when the data are no longer available\n<\/p><p><b>To be interoperable:<\/b>\n<\/p><p>I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.\n<\/p><p>I2. (meta)data use vocabularies that follow FAIR principles\n<\/p><p>I3. (meta)data include qualified references to other (meta)data\n<\/p><p><b>To be reusable:<\/b>\n<\/p><p>R1. meta(data) are richly described with a plurality of accurate and relevant attributes\n<\/p><p>R1.1. (meta)data are released with a clear and accessible data usage license\n<\/p><p>R1.2. (meta)data are associated with detailed provenance\n<\/p><p>R1.3. (meta)data meet domain-relevant community standards\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The elements of the FAIR Principles are related, but independent and separable. The Principles define characteristics that contemporary data resources, tools, vocabularies and infrastructures should exhibit to assist discovery and reuse by third parties. By minimally defining each guiding principle, the barrier-to-entry for data producers, publishers, and stewards who wish to make their data holdings FAIR is purposely maintained as low as possible. The Principles may be adhered to in any combination and incrementally, as data providers\u2019 publishing environments evolve to increasing degrees of \"FAIRness.\" Moreover, the modularity of the Principles, and their distinction between data and metadata, explicitly support a wide range of special circumstances. One such example is highly sensitive or personally identifiable data, where publication of rich metadata to facilitate discovery, including clear rules regarding the process for accessing the data, provides a high degree of \"FAIRness\" even in the absence of FAIR publication of the data itself. A second example involves the publication of non-data research objects. Analytical workflows, for example, are a critical component of the scholarly ecosystem, and their formal publication is necessary to achieve both transparency and scientific reproducibility. The FAIR principles can equally be applied to these non-data assets, which need to be identified, described, discovered, and reused in much the same manner as data.\n<\/p><p>Specific exemplar efforts that provide varying levels of FAIRness are detailed later in this document. Additional issues, however, remain to be addressed. First, when community-endorsed vocabularies or other (meta)data standards do not include the attributes necessary to achieve rich annotation, there are two possible solutions: either publish an extension of an existing, closely related vocabulary, or \u2014 in the extreme case \u2014 create and explicitly publish a new vocabulary resource, following FAIR principles (\"I2\"). Second, for explicitly identifying the standard chosen when more than one vocabulary or other (meta)data standard is available, and given that for instance in the life sciences there are over 600 content standards, the BioSharing registry (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/biosharing.org\/\" target=\"_blank\">https:\/\/biosharing.org\/<\/a>) can be of use as it describes the standards in detail, including versions where applicable.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_Principles_precede_implementation\">The Principles precede implementation<\/span><\/h3>\n<p>These high-level FAIR Guiding Principles precede implementation choices and do not suggest any specific technology, standard, or implementation-solution; moreover, the Principles are not, themselves, a standard or a specification. They act as a guide to data publishers and stewards to assist them in evaluating whether their particular implementation choices are rendering their digital research artifacts findable, accessible, interoperable, and reusable. We anticipate that these high-level principles will enable a broad range of integrative and exploratory behaviors, based on a wide range of technology choices and implementations. Indeed, many repositories are already implementing various aspects of FAIR using a variety of technology choices, and several examples are detailed in the next section; examples include scientific data itself and how narrative data articles are anchored to a progressively FAIR structured metadata.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Examples_of_FAIRness.2C_and_the_resulting_value-added_Dataverse\">Examples of FAIRness, and the resulting value-added Dataverse<\/span><\/h3>\n<p>Dataverse<sup id=\"rdp-ebb-cite_ref-CrosasTheDataverse11_7-1\" class=\"reference\"><a href=\"#cite_note-CrosasTheDataverse11-7\" rel=\"external_link\">[7]<\/a><\/sup> is an open-source data repository software installed in dozens of institutions globally to support public community repositories or institutional research data repositories. Harvard Dataverse, with more than 60,000 datasets, is the largest of the current Dataverse repositories and is open to all researchers from all research fields. Dataverse generates a formal citation for each deposit, following the standard defined by Altman and King.<sup id=\"rdp-ebb-cite_ref-AltmanAProposed07_13-0\" class=\"reference\"><a href=\"#cite_note-AltmanAProposed07-13\" rel=\"external_link\">[13]<\/a><\/sup> Dataverse makes the Digital Object Identifier (DOI) or other persistent identifiers (Handles) public when the dataset is published (\"F\"). This resolves to a landing page, providing access to metadata, data files, dataset terms, waivers or licenses, and version information, all of which is indexed and searchable (\"F,\" \"A,\" and \"R\"). Deposits include metadata, data files, and any complementary files (such as documentation or code) needed to understand the data and analysis (\"R\"). Metadata is always public, even if the data are restricted or removed for privacy issues (\"F,\" \"A\"). This metadata is offered at three levels, extensively supporting the \"I\" and \"R\" FAIR principles: 1) data citation metadata, which maps to DataCite schema or Dublin Core Terms; 2) domain-specific metadata, which when possible maps to metadata standards used within a scientific domain; and 3) file-level metadata, which can be deep and extensive for tabular data files (including column-level metadata). Finally, Dataverse provides public machine-accessible interfaces to search the data, access the metadata, and download the data files, using a token to grant access when data files are restricted (\"A\").\n<\/p><p><b>FAIRDOM<\/b> (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/fair-dom.org\/about\" target=\"_blank\">http:\/\/fair-dom.org\/about<\/a>): integrates the SEEK<sup id=\"rdp-ebb-cite_ref-WolstencroftSEEK15_14-0\" class=\"reference\"><a href=\"#cite_note-WolstencroftSEEK15-14\" rel=\"external_link\">[14]<\/a><\/sup> and <a href=\"https:\/\/www.limswiki.org\/index.php\/OpenBIS\" title=\"OpenBIS\" target=\"_blank\" class=\"wiki-link\" data-key=\"3bbca2b6be0c273677fc99a9474e49ec\">openBIS<\/a><sup id=\"rdp-ebb-cite_ref-BauchOpenBIS11_15-0\" class=\"reference\"><a href=\"#cite_note-BauchOpenBIS11-15\" rel=\"external_link\">[15]<\/a><\/sup> platforms to produce a FAIR data and model management facility for Systems Biology. Individual research assets (or aggregates of data and models) are identified with unique and persistent HTTP URLs, which can be registered with DOIs for publication (\"F\"). Assets can be accessed over the internet in a variety of formats appropriate for individuals and\/or their computers (RDF, XML) (\"I\"). Research assets are annotated with rich metadata, using community standards, formats and ontologies (\"I\"). The metadata is stored as RDF to enable interoperability and assets can be downloaded for reuse (\"R\").\n<\/p><p><b>ISA<\/b>: is a community-driven metadata tracking framework to facilitate standards-compliant collection, curation, management and reuse of life science datasets.<sup id=\"rdp-ebb-cite_ref-SansoneToward12_16-0\" class=\"reference\"><a href=\"#cite_note-SansoneToward12-16\" rel=\"external_link\">[16]<\/a><\/sup> ISA provides progressively FAIR structured metadata to <i>Nature Scientific Data<\/i>\u2019s Data Descriptor articles, and many GigaScience data papers, and underpins the EBI MetaboLights database among other data resources. At the core is a general-purpose, extensible ISA model, originally only available as a tabular representation but subsequently enhanced as an RDF-based representation<sup id=\"rdp-ebb-cite_ref-LamNear14_17-0\" class=\"reference\"><a href=\"#cite_note-LamNear14-17\" rel=\"external_link\">[17]<\/a><\/sup>, and JSON serializations to enable the \"I\" and \"R,\" becoming \"FAIR\" when published as linked data (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/elixir-uk.org\/node-events\/201cisa-as-a-fair-research-object201d-hack-the-spec-event-1\" target=\"_blank\">http:\/\/elixir-uk.org\/node-events\/201cisa-as-a-fair-research-object201d-hack-the-spec-event-1<\/a>) and complementing other research objects.<sup id=\"rdp-ebb-cite_ref-Gonz.C3.A1lez-Beltr.C3.A1nFromPeer15_18-0\" class=\"reference\"><a href=\"#cite_note-Gonz.C3.A1lez-Beltr.C3.A1nFromPeer15-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/p><p><b>Open PHACTS<\/b>: Open PHACTS<sup id=\"rdp-ebb-cite_ref-HarlandOpen12_19-0\" class=\"reference\"><a href=\"#cite_note-HarlandOpen12-19\" rel=\"external_link\">[19]<\/a><\/sup> is a data integration platform for information pertaining to drug discovery. Access to the platform is mediated through a machine-accessible interface<sup id=\"rdp-ebb-cite_ref-GrothAPI14_20-0\" class=\"reference\"><a href=\"#cite_note-GrothAPI14-20\" rel=\"external_link\">[20]<\/a><\/sup> which provides multiple representations that are both human (HTML) and machine readable (RDF, JSON, XML, CSV, etc.), providing the \"A\" facet of FAIRness. The interface allows multiple URLs to be used to access information about a particular entity through a mapping service (\"F\" and \"A\"). Thus, a user can provide a ChEMBL URL to retrieve information sourced from, for example, Chemspider or DrugBank. Each call provides a canonical URL in its response (\"A\" and \"I\"). All data sources used are described using standardized dataset descriptions, following the global VoID standard, with rich provenance (\"R\" and \"I\"). All interface features are described using RDF following the Linked Data API specification (\"A\"). Finally, a majority of the datasets are described using community agreed upon ontologies (\"I\").\n<\/p><p><b>wwPDB<\/b>: wwPDB<sup id=\"rdp-ebb-cite_ref-BermanAnnoun03_4-1\" class=\"reference\"><a href=\"#cite_note-BermanAnnoun03-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BermanTheProt00_21-0\" class=\"reference\"><a href=\"#cite_note-BermanTheProt00-21\" rel=\"external_link\">[21]<\/a><\/sup> is a special-purpose, intensively curated data archive that hosts information about experimentally determined 3D structures of proteins and nucleic acids. All wwPDB entries are stably hosted on an FTP server (\"A\") and represented in machine-readable formats (text and XML); the latter are machine-actionable using the metadata provided by the wwPDB conforming to the Macromolecular Information Framework (mmCIF)<sup id=\"rdp-ebb-cite_ref-BourneMacro97_22-0\" class=\"reference\"><a href=\"#cite_note-BourneMacro97-22\" rel=\"external_link\">[22]<\/a><\/sup>, a data standard of the International Union of Crystallography (IUCr) (\"F,\" \"I\" for humans and IUCr-aware machines). The wwPDB metadata contains cross-references to common identifiers such as PubMed and NCBI Taxonomy, and their wwPDB metadata are described in data dictionaries and schema documents (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/mmcif.wwpdb.org\" target=\"_blank\">http:\/\/mmcif.wwpdb.org<\/a> and <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/pdbml.wwpdb.org\" target=\"_blank\">http:\/\/pdbml.wwpdb.org<\/a>) which conform to the IUCr data standard for the chemical and structural biology domains (\"R\"). A variety of software tools are available to interpret both wwPDB data and meta-data (\"I,\" \"R\" for humans and machines with this software). Each entry is represented by a DOI (\"F,\" \"A\" for humans and machines). The DOI resolves to a zipped file which requires special software for further interrogation\/interpretation. Other wwPDB access points<sup id=\"rdp-ebb-cite_ref-RoseTheRCSB15_23-0\" class=\"reference\"><a href=\"#cite_note-RoseTheRCSB15-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KinjoProtein12_24-0\" class=\"reference\"><a href=\"#cite_note-KinjoProtein12-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GutmanasPDBe14_25-0\" class=\"reference\"><a href=\"#cite_note-GutmanasPDBe14-25\" rel=\"external_link\">[25]<\/a><\/sup> provide access to wwPDB records through URLs that are likely to be stable in the long-term (\"F\"), and all data and metadata is searchable through one or more of the wwPDB-affiliated websites (\"F\").\n<\/p><p><b>UniProt<\/b>: UniProt<sup id=\"rdp-ebb-cite_ref-UniProtUniProt15_5-1\" class=\"reference\"><a href=\"#cite_note-UniProtUniProt15-5\" rel=\"external_link\">[5]<\/a><\/sup> is a comprehensive resource for protein sequence and annotation data. All entries are uniquely identified by a stable URL, that provides access to the record in a variety of formats, including a web page, plain-text, and RDF (\"F\" and \"A\"). The record contains rich metadata (\"F\") that is both human-readable (HTML) and machine-readable (text and RDF), where the RDF-formatted response utilizes shared vocabularies and ontologies such as UniProt Core, FALDO, and ECO (\"I\"). Interlinking with more than 150 different databases, every UniProt record has extensive links into, for example, PubMed, enabling rich citation. These links are machine-actionable in the RDF representation (\"R\"). Finally, in the RDF representation, the UniProt Core Ontology explicitly types all records, leaving no ambiguity \u2014 neither for humans nor machines \u2014 about what the data represents (\"R\"), enabling fully-automated retrieval of records and cross-referencing information.\n<\/p><p>In addition to, and in support of, communities and resources that are already pursuing FAIR objectives, the Data Citation Implementation Group of Force11 has published specific technical recommendations for how to implement many of the principles<sup id=\"rdp-ebb-cite_ref-StarrAchieving15_26-0\" class=\"reference\"><a href=\"#cite_note-StarrAchieving15-26\" rel=\"external_link\">[26]<\/a><\/sup>, with a particular focus on identifiers and their resolution, persistence, and metadata accessibility especially related to citation. In addition, the \"Skunkworks\" group that emerged from the Lorentz Workshop has been creating software supporting infrastructures<sup id=\"rdp-ebb-cite_ref-WilkinsonDataFairPort15_27-0\" class=\"reference\"><a href=\"#cite_note-WilkinsonDataFairPort15-27\" rel=\"external_link\">[27]<\/a><\/sup> that are, end-to-end, compatible with FAIR principles, and can be implemented over existing repositories. These code modules have a particular focus on metadata publication and searchability, compatibility in cases of strict privacy considerations, and the extremely difficult problem of data and metadata interoperability (manuscript in preparation). Finally, there are several emergent projects, some listed in Box 3, for which FAIR is a key objective. These projects may provide valuable advice and guidance for those wishing to become more FAIR.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"1\"><b>Box 3: Emergent community\/collaborative initiatives with FAIR as a core focus or activity<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>bioCADDIE<\/b> (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/biocaddie.org\" target=\"_blank\">https:\/\/biocaddie.org<\/a>): The NIH BD2K biomedical and healthCAre Data Discovery Index Ecosystem (bioCADDIE) consortium works to develop a Data Discovery Index (DDI) prototype, which is set to be as transformative and impactful for data as PubMed for biomedical literature.<sup id=\"rdp-ebb-cite_ref-Ohno-Machado_bioCADDIE15_28-0\" class=\"reference\"><a href=\"#cite_note-Ohno-Machado_bioCADDIE15-28\" rel=\"external_link\">[28]<\/a><\/sup> The DDI focuses on finding (\"F\") and accessing (\"A\") the datasets stored across different sources, and it progressively works to identify relevant metadata<sup id=\"rdp-ebb-cite_ref-WG3WG3_15_29-0\" class=\"reference\"><a href=\"#cite_note-WG3WG3_15-29\" rel=\"external_link\">[29]<\/a><\/sup> (\"I\") and maps them to community standards (\"R\"), linking to BioSharing.\n<p><b>CEDAR<\/b>: The Center for Expanded Data Annotation and Retrieval (CEDAR)<sup id=\"rdp-ebb-cite_ref-MusenTheCenter15_30-0\" class=\"reference\"><a href=\"#cite_note-MusenTheCenter15-30\" rel=\"external_link\">[30]<\/a><\/sup> is an NIH BD2K funded center of excellence to develop tools and technologies that reduce the burden of authoring and enhancing metadata that meet community-based standards. CEDAR will enable the creation of metadata templates that implement community-based standards for experimental metadata, from BioSharing (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/biosharing.org\" target=\"_blank\">https:\/\/biosharing.org<\/a>), and that will be uniquely identifiable and retrievable with HTTP URIs, and annotated with vocabularies and ontologies drawn from BioPortal (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/bioportal.bioontology.org\" target=\"_blank\">http:\/\/bioportal.bioontology.org<\/a>) (\"F,\" \"A,\" \"I,\" \"R\"). These templates will guide users to create rich metadata with unique and stable HTTP identifiers (\"F\") that can be retrieved using HTTP (\"A\") and accessible in a variety of formats (JSON-LD, TURTLE, RDF\/XML, CSV, etc.) (\"I\"). These metadata will use community standards, as defined by the template, and include provenance and data usage (\"R\").\n<\/p><p>These two projects, among others, provide tools and\/or collaborative opportunities for those who wish to improve the FAIRness of their data.\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"FAIRness_is_a_prerequisite_for_proper_data_management_and_data_stewardship\">FAIRness is a prerequisite for proper data management and data stewardship<\/span><\/h3>\n<p>The ideas within the FAIR Guiding Principles reflect, combine, build upon, and extend previous work by both the Concept Web Alliance (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/conceptweblog.wordpress.com\/\" target=\"_blank\">https:\/\/conceptweblog.wordpress.com\/<\/a>) partners, who focused on machine-actionability and harmonization of data structures and semantics, and by the scientific and scholarly organizations that developed the Joint Declaration of Data Citation Principles (JDDCP)<sup id=\"rdp-ebb-cite_ref-DCSGJoint14_31-0\" class=\"reference\"><a href=\"#cite_note-DCSGJoint14-31\" rel=\"external_link\">[31]<\/a><\/sup>, who focused on primary scholarly data being made citable, discoverable, and available for reuse, so as to be capable of supporting more rigorous scholarship. An attempt to define the similarities and overlaps between the FAIR Principles and the JDDCP is provided at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.force11.org\/node\/6062\" target=\"_blank\">https:\/\/www.force11.org\/node\/6062<\/a>. The FAIR Principles are also complementary to the Data Seal of Approval (DSA) (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/datasealofapproval.org\/media\/filer_public\/2013\/09\/27\/guidelines_2014-2015.pdf\" target=\"_blank\">http:\/\/datasealofapproval.org\/media\/filer_public\/2013\/09\/27\/guidelines_2014-2015.pdf<\/a>) in that they share the general aim to render data re-usable for users other than those who originally generated them. While the DSA focuses primarily on the responsibilities and conduct of data producers and repositories, FAIR focuses primarily on the data itself. Clearly, the broader community of stakeholders is coalescing around a set of common, dovetailed visions spanning all facets of the scholarly data publishing ecosystem.\n<\/p><p>The end result, when implemented, will be more rigorous management and stewardship of these valuable digital resources, to the benefit of the entire academic community. As stated at the outset, good data management and stewardship is not a goal in itself, but rather a precondition supporting knowledge discovery and innovation. Contemporary e-Science requires data to be findable, accessible, interoperable, and reusable in the long-term, and these objectives are rapidly becoming expectations of agencies and publishers. We demonstrate, therefore, that the FAIR Data Principles provide a set of mileposts for data producers and publishers. They guide the implementation of the most basic levels of good Data Management and Stewardship practice, thus helping researchers adhere to the expectations and requirements of their funding agencies. We call on all data producers and publishers to examine and implement these principles, and actively participate with the FAIR initiative by joining the Force11 working group. By working together towards shared, common goals, the valuable data produced by our community will gradually achieve the critical goals of FAIRness.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The original Lorentz Workshop \"Jointly Designing a Data FAIRport\" was organized by Barend Mons in collaboration with and co-sponsored by the Lorentz center, The Dutch Techcenter for the Life Sciences, and the Netherlands eScience Center. The principles and themes described in this manuscript represent the significant voluntary contributions and participation of the authors at, and\/or subsequent to, this workshop and from the wider FORCE11, BD2K, and ELIXIR communities. We also acknowledge and thank the organizers and backers of the NBDC\/DBCLS BioHackathon 2015, where several of the authors made significant revisions to the FAIR Principles.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Author_information\">Author information<\/span><\/h2>\n<p>M.W. was the primary author of the manuscript and participated extensively in the drafting and editing of the FAIR Principles. M.D. was significantly involved in the drafting of the FAIR Principles. B.M. conceived of the FAIR Data Initiative and contributed extensively to the drafting of the principles and this manuscript text. All other authors are listed alphabetically, and contributed to the manuscript either by their participation in the initial workshop and\/or by editing or commenting on the manuscript text.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>M.A. is <i>Nature Genetics<\/i>\u2019 Editor in Chief; S.A.S. is <i>Scientific Data<\/i>\u2019s Honorary Academic Editor and consultant.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-RochePublic15-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RochePublic15_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-RochePublic15_1-1\" rel=\"external_link\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Roche, D.G.; Kruuk, L.E.; Lanfear, R.; Binning, S.A. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4640582\" target=\"_blank\">\"Public data archiving in ecology and evolution: How well are we doing?\"<\/a>. <i>PLOS Biology<\/i> <b>13<\/b>: e1002295. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pbio.1002295\" target=\"_blank\">10.1371\/journal.pbio.1002295<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4640582\/\" target=\"_blank\">PMC4640582<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26556502\" target=\"_blank\">26556502<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4640582\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4640582<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Public+data+archiving+in+ecology+and+evolution%3A+How+well+are+we+doing%3F&rft.jtitle=PLOS+Biology&rft.aulast=Roche%2C+D.G.%3B+Kruuk%2C+L.E.%3B+Lanfear%2C+R.%3B+Binning%2C+S.A.&rft.au=Roche%2C+D.G.%3B+Kruuk%2C+L.E.%3B+Lanfear%2C+R.%3B+Binning%2C+S.A.&rft.date=2015&rft.volume=13&rft.pages=e1002295&rft_id=info:doi\/10.1371%2Fjournal.pbio.1002295&rft_id=info:pmc\/PMC4640582&rft_id=info:pmid\/26556502&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4640582&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BechhoferResearch10-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BechhoferResearch10_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bechhofer, S.; De Roure, D.; Gamble, M. et al. (2010). \"Research objects: Towards exchange and reuse of digital knowledge\". <i>Nature Precedings<\/i>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnpre.2010.4626.1\" target=\"_blank\">10.1038\/npre.2010.4626.1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Research+objects%3A+Towards+exchange+and+reuse+of+digital+knowledge&rft.jtitle=Nature+Precedings&rft.aulast=Bechhofer%2C+S.%3B+De+Roure%2C+D.%3B+Gamble%2C+M.+et+al.&rft.au=Bechhofer%2C+S.%3B+De+Roure%2C+D.%3B+Gamble%2C+M.+et+al.&rft.date=2010&rft_id=info:doi\/10.1038%2Fnpre.2010.4626.1&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BensonGenBank13-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BensonGenBank13_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Benson, D.A.; Cavanaugh, M.; Clark, K. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4640582\" target=\"_blank\">\"GenBank\"<\/a>. <i>Nucleic Acids Research<\/i> <b>41<\/b> (D1): D36-42. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgks1195\" target=\"_blank\">10.1093\/nar\/gks1195<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4640582\/\" target=\"_blank\">PMC4640582<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/PMC3531190\" target=\"_blank\">PMC3531190<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4640582\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4640582<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GenBank&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Benson%2C+D.A.%3B+Cavanaugh%2C+M.%3B+Clark%2C+K.+et+al.&rft.au=Benson%2C+D.A.%3B+Cavanaugh%2C+M.%3B+Clark%2C+K.+et+al.&rft.date=2013&rft.volume=41&rft.issue=D1&rft.pages=D36-42&rft_id=info:doi\/10.1093%2Fnar%2Fgks1195&rft_id=info:pmc\/PMC4640582&rft_id=info:pmid\/PMC3531190&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4640582&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BermanAnnoun03-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BermanAnnoun03_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-BermanAnnoun03_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berman, H.; Henrick, K.; Nakamura, H. (2003). \"Announcing the worldwide Protein Data Bank\". <i>Nature Structural Biology<\/i> <b>10<\/b> (12): 980. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnsb1203-980\" target=\"_blank\">10.1038\/nsb1203-980<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/14634627\" target=\"_blank\">14634627<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Announcing+the+worldwide+Protein+Data+Bank&rft.jtitle=Nature+Structural+Biology&rft.aulast=Berman%2C+H.%3B+Henrick%2C+K.%3B+Nakamura%2C+H.&rft.au=Berman%2C+H.%3B+Henrick%2C+K.%3B+Nakamura%2C+H.&rft.date=2003&rft.volume=10&rft.issue=12&rft.pages=980&rft_id=info:doi\/10.1038%2Fnsb1203-980&rft_id=info:pmid\/14634627&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UniProtUniProt15-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-UniProtUniProt15_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-UniProtUniProt15_5-1\" rel=\"external_link\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">UniProt Consortium (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4384041\" target=\"_blank\">\"UniProt: A hub for protein information\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (D1): D204-12. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku989\" target=\"_blank\">10.1093\/nar\/gku989<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4384041\/\" target=\"_blank\">PMC4384041<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25348405\" target=\"_blank\">25348405<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4384041\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4384041<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=UniProt%3A+A+hub+for+protein+information&rft.jtitle=Nucleic+Acids+Research&rft.aulast=UniProt+Consortium&rft.au=UniProt+Consortium&rft.date=2015&rft.volume=43&rft.issue=D1&rft.pages=D204-12&rft_id=info:doi\/10.1093%2Fnar%2Fgku989&rft_id=info:pmc\/PMC4384041&rft_id=info:pmid\/25348405&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4384041&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WengerTheSIMBAD00-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WengerTheSIMBAD00_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wenger, M.; Ochsenbein, F.; Egret, D. et al. (2000). \"The SIMBAD astronomical database: The CDS reference database for astronomical objects\". <i>Astronomy and Astrophysics Supplement Series<\/i> <b>143<\/b> (1): 9\u201322. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1051%2Faas%3A2000332\" target=\"_blank\">10.1051\/aas:2000332<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+SIMBAD+astronomical+database%3A+The+CDS+reference+database+for+astronomical+objects&rft.jtitle=Astronomy+and+Astrophysics+Supplement+Series&rft.aulast=Wenger%2C+M.%3B+Ochsenbein%2C+F.%3B+Egret%2C+D.+et+al.&rft.au=Wenger%2C+M.%3B+Ochsenbein%2C+F.%3B+Egret%2C+D.+et+al.&rft.date=2000&rft.volume=143&rft.issue=1&rft.pages=9%E2%80%9322&rft_id=info:doi\/10.1051%2Faas%3A2000332&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CrosasTheDataverse11-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CrosasTheDataverse11_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-CrosasTheDataverse11_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Crosas, M. (2011). \"The Dataverse Network: An open-source application for sharing, discovering and preserving data\". <i>D-Lib Magazine<\/i> <b>17<\/b> (1\/2): 2. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1045%2Fjanuary2011-crosas\" target=\"_blank\">10.1045\/january2011-crosas<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Dataverse+Network%3A+An+open-source+application+for+sharing%2C+discovering+and+preserving+data&rft.jtitle=D-Lib+Magazine&rft.aulast=Crosas%2C+M.&rft.au=Crosas%2C+M.&rft.date=2011&rft.volume=17&rft.issue=1%2F2&rft.pages=2&rft_id=info:doi\/10.1045%2Fjanuary2011-crosas&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WhiteTheDryad08-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WhiteTheDryad08_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">White, H.C.; Carrier, S.; Thompson, A. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dcpapers.dublincore.org\/pubs\/article\/view\/928\" target=\"_blank\">\"The Dryad Data Repository: A Singapore Framework metadata architecture in a DSpace environment\"<\/a>. <i>DC-2008--Berlin Proceedings<\/i> <b>2008<\/b>: 157\u2013162<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dcpapers.dublincore.org\/pubs\/article\/view\/928\" target=\"_blank\">http:\/\/dcpapers.dublincore.org\/pubs\/article\/view\/928<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Dryad+Data+Repository%3A+A+Singapore+Framework+metadata+architecture+in+a+DSpace+environment&rft.jtitle=DC-2008--Berlin+Proceedings&rft.aulast=White%2C+H.C.%3B+Carrier%2C+S.%3B+Thompson%2C+A.+et+al.&rft.au=White%2C+H.C.%3B+Carrier%2C+S.%3B+Thompson%2C+A.+et+al.&rft.date=2008&rft.volume=2008&rft.pages=157%E2%80%93162&rft_id=http%3A%2F%2Fdcpapers.dublincore.org%2Fpubs%2Farticle%2Fview%2F928&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LecarpentierEUDAT13-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LecarpentierEUDAT13_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lecarpentier, D.; Wittenburg, P.; Elbers, W. (2013). \"EUDAT: A new cross-disciplinary data infrastructure for science\". <i>International Journal of Digital Curation<\/i> <b>8<\/b> (1): 279\u2013287. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2218%2Fijdc.v8i1.260\" target=\"_blank\">10.2218\/ijdc.v8i1.260<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EUDAT%3A+A+new+cross-disciplinary+data+infrastructure+for+science&rft.jtitle=International+Journal+of+Digital+Curation&rft.aulast=Lecarpentier%2C+D.%3B+Wittenburg%2C+P.%3B+Elbers%2C+W.&rft.au=Lecarpentier%2C+D.%3B+Wittenburg%2C+P.%3B+Elbers%2C+W.&rft.date=2013&rft.volume=8&rft.issue=1&rft.pages=279%E2%80%93287&rft_id=info:doi\/10.2218%2Fijdc.v8i1.260&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MartoneForce11_15-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MartoneForce11_15_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Martone, M.E. (2015). \"FORCE11: Building the Future for Research Communications and e-Scholarship\". <i>BioScience<\/i> <b>65<\/b> (7): 635. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbiosci%2Fbiv095\" target=\"_blank\">10.1093\/biosci\/biv095<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=FORCE11%3A+Building+the+Future+for+Research+Communications+and+e-Scholarship&rft.jtitle=BioScience&rft.aulast=Martone%2C+M.E.&rft.au=Martone%2C+M.E.&rft.date=2015&rft.volume=65&rft.issue=7&rft.pages=635&rft_id=info:doi\/10.1093%2Fbiosci%2Fbiv095&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WhiteNine13-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WhiteNine13_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">White, E.P.; Baldridge, E.; Brym, Z.T. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ojs.library.queensu.ca\/index.php\/IEE\/article\/view\/4608\/0\" target=\"_blank\">\"Nine simple ways to make it easier to (re)use your data\"<\/a>. <i>Ideas in Ecology and Evolution<\/i> <b>6<\/b> (2)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ojs.library.queensu.ca\/index.php\/IEE\/article\/view\/4608\/0\" target=\"_blank\">http:\/\/ojs.library.queensu.ca\/index.php\/IEE\/article\/view\/4608\/0<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Nine+simple+ways+to+make+it+easier+to+%28re%29use+your+data&rft.jtitle=Ideas+in+Ecology+and+Evolution&rft.aulast=White%2C+E.P.%3B+Baldridge%2C+E.%3B+Brym%2C+Z.T.+et+al.&rft.au=White%2C+E.P.%3B+Baldridge%2C+E.%3B+Brym%2C+Z.T.+et+al.&rft.date=2013&rft.volume=6&rft.issue=2&rft_id=http%3A%2F%2Fojs.library.queensu.ca%2Findex.php%2FIEE%2Farticle%2Fview%2F4608%2F0&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SandveTen13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SandveTen13_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sandve, G.K.; Nekrutenko, A.; Taylor, J.; Hovig, E. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3812051\" target=\"_blank\">\"Ten simple rules for reproducible computational research\"<\/a>. <i>PLOS Computational Biology<\/i> <b>9<\/b> (10). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1003285\" target=\"_blank\">10.1371\/journal.pcbi.1003285<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3812051\/\" target=\"_blank\">PMC3812051<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24204232\" target=\"_blank\">24204232<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3812051\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3812051<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ten+simple+rules+for+reproducible+computational+research&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Sandve%2C+G.K.%3B+Nekrutenko%2C+A.%3B+Taylor%2C+J.%3B+Hovig%2C+E.&rft.au=Sandve%2C+G.K.%3B+Nekrutenko%2C+A.%3B+Taylor%2C+J.%3B+Hovig%2C+E.&rft.date=2013&rft.volume=9&rft.issue=10&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1003285&rft_id=info:pmc\/PMC3812051&rft_id=info:pmid\/24204232&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3812051&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AltmanAProposed07-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AltmanAProposed07_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Altman, M.; King, G. (2007). \"A proposed standard for the scholarly citation of quantitative data\". <i>D-Lib Magazine<\/i> <b>13<\/b> (3\/4). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1045%2Fmarch2007-altman\" target=\"_blank\">10.1045\/march2007-altman<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+proposed+standard+for+the+scholarly+citation+of+quantitative+data&rft.jtitle=D-Lib+Magazine&rft.aulast=Altman%2C+M.%3B+King%2C+G.&rft.au=Altman%2C+M.%3B+King%2C+G.&rft.date=2007&rft.volume=13&rft.issue=3%2F4&rft_id=info:doi\/10.1045%2Fmarch2007-altman&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WolstencroftSEEK15-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WolstencroftSEEK15_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wolstencroft, K.; Owen, S.; Krebs, O. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4702362\" target=\"_blank\">\"SEEK: A systems biology data and model management platform\"<\/a>. <i>BMC Systems Biology<\/i> <b>9<\/b>: 33. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12918-015-0174-y\" target=\"_blank\">10.1186\/s12918-015-0174-y<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4702362\/\" target=\"_blank\">PMC4702362<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26160520\" target=\"_blank\">26160520<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4702362\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4702362<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SEEK%3A+A+systems+biology+data+and+model+management+platform&rft.jtitle=BMC+Systems+Biology&rft.aulast=Wolstencroft%2C+K.%3B+Owen%2C+S.%3B+Krebs%2C+O.+et+al.&rft.au=Wolstencroft%2C+K.%3B+Owen%2C+S.%3B+Krebs%2C+O.+et+al.&rft.date=2015&rft.volume=9&rft.pages=33&rft_id=info:doi\/10.1186%2Fs12918-015-0174-y&rft_id=info:pmc\/PMC4702362&rft_id=info:pmid\/26160520&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4702362&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BauchOpenBIS11-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BauchOpenBIS11_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bauch, A.; Adamczyk, I.; Buczek, P. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639\" target=\"_blank\">\"openBIS: A flexible framework for managing and analyzing complex data in biology research\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b>: 468. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-468\" target=\"_blank\">10.1186\/1471-2105-12-468<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3275639\/\" target=\"_blank\">PMC3275639<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22151573\" target=\"_blank\">22151573<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=openBIS%3A+A+flexible+framework+for+managing+and+analyzing+complex+data+in+biology+research&rft.jtitle=BMC+Bioinformatics&rft.aulast=Bauch%2C+A.%3B+Adamczyk%2C+I.%3B+Buczek%2C+P.+et+al.&rft.au=Bauch%2C+A.%3B+Adamczyk%2C+I.%3B+Buczek%2C+P.+et+al.&rft.date=2011&rft.volume=12&rft.pages=468&rft_id=info:doi\/10.1186%2F1471-2105-12-468&rft_id=info:pmc\/PMC3275639&rft_id=info:pmid\/22151573&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3275639&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SansoneToward12-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SansoneToward12_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sansone, S.A.; Rocca-Serra, P.; Field, D. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3428019\" target=\"_blank\">\"Toward interoperable bioscience data\"<\/a>. <i>Nature Genetics<\/i> <b>44<\/b> (2): 121\u20136. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fng.1054\" target=\"_blank\">10.1038\/ng.1054<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3428019\/\" target=\"_blank\">PMC3428019<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22281772\" target=\"_blank\">22281772<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3428019\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3428019<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+interoperable+bioscience+data&rft.jtitle=Nature+Genetics&rft.aulast=Sansone%2C+S.A.%3B+Rocca-Serra%2C+P.%3B+Field%2C+D.+et+al.&rft.au=Sansone%2C+S.A.%3B+Rocca-Serra%2C+P.%3B+Field%2C+D.+et+al.&rft.date=2012&rft.volume=44&rft.issue=2&rft.pages=121%E2%80%936&rft_id=info:doi\/10.1038%2Fng.1054&rft_id=info:pmc\/PMC3428019&rft_id=info:pmid\/22281772&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3428019&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LamNear14-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LamNear14_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lam, K.K.; Khalak, A.; Tse, D. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4168708\" target=\"_blank\">\"Near-optimal assembly for shotgun sequencing with noisy reads\"<\/a>. <i>BMC Bioinformatics<\/i> <b>15<\/b> (Supp. 9): S4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-15-S9-S4\" target=\"_blank\">10.1186\/1471-2105-15-S9-S4<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4168708\/\" target=\"_blank\">PMC4168708<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25252708\" target=\"_blank\">25252708<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4168708\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4168708<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Near-optimal+assembly+for+shotgun+sequencing+with+noisy+reads&rft.jtitle=BMC+Bioinformatics&rft.aulast=Lam%2C+K.K.%3B+Khalak%2C+A.%3B+Tse%2C+D.&rft.au=Lam%2C+K.K.%3B+Khalak%2C+A.%3B+Tse%2C+D.&rft.date=2014&rft.volume=15&rft.issue=Supp.+9&rft.pages=S4&rft_id=info:doi\/10.1186%2F1471-2105-15-S9-S4&rft_id=info:pmc\/PMC4168708&rft_id=info:pmid\/25252708&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4168708&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Gonz.C3.A1lez-Beltr.C3.A1nFromPeer15-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Gonz.C3.A1lez-Beltr.C3.A1nFromPeer15_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gonz\u00e1lez-Beltr\u00e1n, A.; Li, P.; Zhao, J. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4495984\" target=\"_blank\">\"From peer-reviewed to peer-reproduced in scholarly publishing: The complementary roles of data models and workflows in bioinformatics\"<\/a>. <i>PLOS One<\/i> <b>10<\/b> (7): e0127612. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0127612\" target=\"_blank\">10.1371\/journal.pone.0127612<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4495984\/\" target=\"_blank\">PMC4495984<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26154165\" target=\"_blank\">26154165<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4495984\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4495984<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+peer-reviewed+to+peer-reproduced+in+scholarly+publishing%3A+The+complementary+roles+of+data+models+and+workflows+in+bioinformatics&rft.jtitle=PLOS+One&rft.aulast=Gonz%C3%A1lez-Beltr%C3%A1n%2C+A.%3B+Li%2C+P.%3B+Zhao%2C+J.+et+al.&rft.au=Gonz%C3%A1lez-Beltr%C3%A1n%2C+A.%3B+Li%2C+P.%3B+Zhao%2C+J.+et+al.&rft.date=2015&rft.volume=10&rft.issue=7&rft.pages=e0127612&rft_id=info:doi\/10.1371%2Fjournal.pone.0127612&rft_id=info:pmc\/PMC4495984&rft_id=info:pmid\/26154165&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4495984&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarlandOpen12-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HarlandOpen12_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Harland, L. (2012). \"Open PHACTS: A semantic knowledge infrastructure for public and commercial drug discovery research\". <i>Knowledge Engineering and Knowledge Management - EKAW 2012<\/i> <b>7603<\/b>: 1\u20137. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-33876-2_1\" target=\"_blank\">10.1007\/978-3-642-33876-2_1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+PHACTS%3A+A+semantic+knowledge+infrastructure+for+public+and+commercial+drug+discovery+research&rft.jtitle=Knowledge+Engineering+and+Knowledge+Management+-+EKAW+2012&rft.aulast=Harland%2C+L.&rft.au=Harland%2C+L.&rft.date=2012&rft.volume=7603&rft.pages=1%E2%80%937&rft_id=info:doi\/10.1007%2F978-3-642-33876-2_1&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GrothAPI14-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GrothAPI14_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Groth, P.; Loizou, A.; Gray, A.J.G. et al. (2014). \"API-centric Linked Data integration: The Open PHACTS Discovery Platform case study\". <i>Web Semantics: Science, Services and Agents on the World Wide Web<\/i> <b>29<\/b>: 12\u201318. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.websem.2014.03.003\" target=\"_blank\">10.1016\/j.websem.2014.03.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=API-centric+Linked+Data+integration%3A+The+Open+PHACTS+Discovery+Platform+case+study&rft.jtitle=Web+Semantics%3A+Science%2C+Services+and+Agents+on+the+World+Wide+Web&rft.aulast=Groth%2C+P.%3B+Loizou%2C+A.%3B+Gray%2C+A.J.G.+et+al.&rft.au=Groth%2C+P.%3B+Loizou%2C+A.%3B+Gray%2C+A.J.G.+et+al.&rft.date=2014&rft.volume=29&rft.pages=12%E2%80%9318&rft_id=info:doi\/10.1016%2Fj.websem.2014.03.003&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BermanTheProt00-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BermanTheProt00_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berman, H.M.; Westbrook, J.; Feng, Z. et al. (2000). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC102472\" target=\"_blank\">\"The Protein Data Bank\"<\/a>. <i>Nucleic Acids Research<\/i> <b>28<\/b> (1): 235-42. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC102472\/\" target=\"_blank\">PMC102472<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10592235\" target=\"_blank\">10592235<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC102472\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC102472<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Protein+Data+Bank&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Berman%2C+H.M.%3B+Westbrook%2C+J.%3B+Feng%2C+Z.+et+al.&rft.au=Berman%2C+H.M.%3B+Westbrook%2C+J.%3B+Feng%2C+Z.+et+al.&rft.date=2000&rft.volume=28&rft.issue=1&rft.pages=235-42&rft_id=info:pmc\/PMC102472&rft_id=info:pmid\/10592235&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC102472&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BourneMacro97-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BourneMacro97_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bourne, P.E.; Berman, H.M.; McMahon, B. et al. (1997). \"Macromolecular Crystallographic Information File\". <i>Methods in Enzymology<\/i> <b>277<\/b>: 571\u201390. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18488325\" target=\"_blank\">18488325<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Macromolecular+Crystallographic+Information+File&rft.jtitle=Methods+in+Enzymology&rft.aulast=Bourne%2C+P.E.%3B+Berman%2C+H.M.%3B+McMahon%2C+B.+et+al.&rft.au=Bourne%2C+P.E.%3B+Berman%2C+H.M.%3B+McMahon%2C+B.+et+al.&rft.date=1997&rft.volume=277&rft.pages=571%E2%80%9390&rft_id=info:pmid\/18488325&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoseTheRCSB15-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RoseTheRCSB15_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rose, P.W.; Prli\u0107, A.; Bi, C. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383988\" target=\"_blank\">\"The RCSB Protein Data Bank: Views of structural biology for basic and applied research and education\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (D1): D345-56. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1214\" target=\"_blank\">10.1093\/nar\/gku1214<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383988\/\" target=\"_blank\">PMC4383988<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25428375\" target=\"_blank\">25428375<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383988\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383988<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+RCSB+Protein+Data+Bank%3A+Views+of+structural+biology+for+basic+and+applied+research+and+education&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Rose%2C+P.W.%3B+Prli%C4%87%2C+A.%3B+Bi%2C+C.+et+al.&rft.au=Rose%2C+P.W.%3B+Prli%C4%87%2C+A.%3B+Bi%2C+C.+et+al.&rft.date=2015&rft.volume=43&rft.issue=D1&rft.pages=D345-56&rft_id=info:doi\/10.1093%2Fnar%2Fgku1214&rft_id=info:pmc\/PMC4383988&rft_id=info:pmid\/25428375&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383988&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KinjoProtein12-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KinjoProtein12_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kinjo, A.R.; Suzuki, H.; Yamashita, R. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3245181\" target=\"_blank\">\"Protein Data Bank Japan (PDBj): Maintaining a structural data archive and resource description framework format\"<\/a>. <i>Nucleic Acids Research<\/i> <b>40<\/b> (D1): D453-60. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkr811\" target=\"_blank\">10.1093\/nar\/gkr811<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3245181\/\" target=\"_blank\">PMC3245181<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21976737\" target=\"_blank\">21976737<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3245181\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3245181<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Protein+Data+Bank+Japan+%28PDBj%29%3A+Maintaining+a+structural+data+archive+and+resource+description+framework+format&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Kinjo%2C+A.R.%3B+Suzuki%2C+H.%3B+Yamashita%2C+R.+et+al.&rft.au=Kinjo%2C+A.R.%3B+Suzuki%2C+H.%3B+Yamashita%2C+R.+et+al.&rft.date=2012&rft.volume=40&rft.issue=D1&rft.pages=D453-60&rft_id=info:doi\/10.1093%2Fnar%2Fgkr811&rft_id=info:pmc\/PMC3245181&rft_id=info:pmid\/21976737&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3245181&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GutmanasPDBe14-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GutmanasPDBe14_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gutmanas, A.; Alhroub, Y.; Battle, G.M. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3965016\" target=\"_blank\">\"PDBe: Protein Data Bank in Europe\"<\/a>. <i>Nucleic Acids Research<\/i> <b>42<\/b> (D1): D285-91. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkt1180\" target=\"_blank\">10.1093\/nar\/gkt1180<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3965016\/\" target=\"_blank\">PMC3965016<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24288376\" target=\"_blank\">24288376<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3965016\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3965016<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PDBe%3A+Protein+Data+Bank+in+Europe&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Gutmanas%2C+A.%3B+Alhroub%2C+Y.%3B+Battle%2C+G.M.+et+al.&rft.au=Gutmanas%2C+A.%3B+Alhroub%2C+Y.%3B+Battle%2C+G.M.+et+al.&rft.date=2014&rft.volume=42&rft.issue=D1&rft.pages=D285-91&rft_id=info:doi\/10.1093%2Fnar%2Fgkt1180&rft_id=info:pmc\/PMC3965016&rft_id=info:pmid\/24288376&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3965016&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StarrAchieving15-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StarrAchieving15_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Starr, J.; Castro, E.; Crosas, M. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4498574\" target=\"_blank\">\"Achieving human and machine accessibility of cited data in scholarly publications\"<\/a>. <i>PeerJ Computer Science<\/i> <b>1<\/b>: e1. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717%2Fpeerj-cs.1\" target=\"_blank\">10.7717\/peerj-cs.1<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4498574\/\" target=\"_blank\">PMC4498574<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26167542\" target=\"_blank\">26167542<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4498574\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4498574<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Achieving+human+and+machine+accessibility+of+cited+data+in+scholarly+publications&rft.jtitle=PeerJ+Computer+Science&rft.aulast=Starr%2C+J.%3B+Castro%2C+E.%3B+Crosas%2C+M.+et+al.&rft.au=Starr%2C+J.%3B+Castro%2C+E.%3B+Crosas%2C+M.+et+al.&rft.date=2015&rft.volume=1&rft.pages=e1&rft_id=info:doi\/10.7717%2Fpeerj-cs.1&rft_id=info:pmc\/PMC4498574&rft_id=info:pmid\/26167542&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4498574&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilkinsonDataFairPort15-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilkinsonDataFairPort15_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wilkinson, M.; Dumontier, M.; Durbin, P. (10 November 2015). \"DataFairPort: The Perl libraries version 0.231\". <i>Zenodo<\/i>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/http%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.33584\" target=\"_blank\">http:\/\/doi.org\/10.5281\/zenodo.33584<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DataFairPort%3A+The+Perl+libraries+version+0.231&rft.jtitle=Zenodo&rft.aulast=Wilkinson%2C+M.%3B+Dumontier%2C+M.%3B+Durbin%2C+P.&rft.au=Wilkinson%2C+M.%3B+Dumontier%2C+M.%3B+Durbin%2C+P.&rft.date=10+November+2015&rft_id=info:doi\/http%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.33584&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Ohno-Machado_bioCADDIE15-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Ohno-Machado_bioCADDIE15_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Ohno-Machado, L.; Alter, G.; Fore, I. et al. (08 March 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/figshare.com\/articles\/bioCADDIE_white_paper_Data_Discovery_Index\/1362572\" target=\"_blank\">\"bioCADDIE White Paper: Vision and Implementation\"<\/a>. <i>figshare<\/i>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.6084%2Fm9.figshare.1362572\" target=\"_blank\">10.6084\/m9.figshare.1362572<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/figshare.com\/articles\/bioCADDIE_white_paper_Data_Discovery_Index\/1362572\" target=\"_blank\">https:\/\/figshare.com\/articles\/bioCADDIE_white_paper_Data_Discovery_Index\/1362572<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=bioCADDIE+White+Paper%3A+Vision+and+Implementation&rft.atitle=figshare&rft.aulast=Ohno-Machado%2C+L.%3B+Alter%2C+G.%3B+Fore%2C+I.+et+al.&rft.au=Ohno-Machado%2C+L.%3B+Alter%2C+G.%3B+Fore%2C+I.+et+al.&rft.date=08+March+2015&rft_id=info:doi\/10.6084%2Fm9.figshare.1362572&rft_id=https%3A%2F%2Ffigshare.com%2Farticles%2FbioCADDIE_white_paper_Data_Discovery_Index%2F1362572&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WG3WG3_15-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WG3WG3_15_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">WG3 Members (14 August 2015). \"WG3-MetadataSpecifications: NIH BD2K bioCADDIE Data Discovery Index WG3 Metadata Specification v1\". <i>Zenodo<\/i>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5281%2Fzenodo.28019\" target=\"_blank\">10.5281\/zenodo.28019<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=WG3-MetadataSpecifications%3A+NIH+BD2K+bioCADDIE+Data+Discovery+Index+WG3+Metadata+Specification+v1&rft.jtitle=Zenodo&rft.aulast=WG3+Members&rft.au=WG3+Members&rft.date=14+August+2015&rft_id=info:doi\/10.5281%2Fzenodo.28019&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MusenTheCenter15-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MusenTheCenter15_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Musen, M.A.; Bean, C.A.; Cheung, K.H. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5009916\" target=\"_blank\">\"The center for expanded data annotation and retrieval\"<\/a>. <i>JAMIA<\/i> <b>22<\/b> (6): 1148-52. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fjamia%2Focv048\" target=\"_blank\">10.1093\/jamia\/ocv048<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5009916\/\" target=\"_blank\">PMC5009916<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26112029\" target=\"_blank\">26112029<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5009916\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5009916<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+center+for+expanded+data+annotation+and+retrieval&rft.jtitle=JAMIA&rft.aulast=Musen%2C+M.A.%3B+Bean%2C+C.A.%3B+Cheung%2C+K.H.+et+al.&rft.au=Musen%2C+M.A.%3B+Bean%2C+C.A.%3B+Cheung%2C+K.H.+et+al.&rft.date=2015&rft.volume=22&rft.issue=6&rft.pages=1148-52&rft_id=info:doi\/10.1093%2Fjamia%2Focv048&rft_id=info:pmc\/PMC5009916&rft_id=info:pmid\/26112029&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5009916&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DCSGJoint14-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DCSGJoint14_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Data Citation Synthesis Group (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.force11.org\/datacitation\" target=\"_blank\">\"Joint Declaration of Data Citation Principles\"<\/a>. In Martone, M.. FORCE11<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.force11.org\/datacitation\" target=\"_blank\">https:\/\/www.force11.org\/datacitation<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Joint+Declaration+of+Data+Citation+Principles&rft.atitle=&rft.aulast=Data+Citation+Synthesis+Group&rft.au=Data+Citation+Synthesis+Group&rft.date=2014&rft.pub=FORCE11&rft_id=https%3A%2F%2Fwww.force11.org%2Fdatacitation&rfr_id=info:sid\/en.wikipedia.org:Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. In the original, reference 26 accidentally duplicates reference five; 26 was omitted and five used in its place. The original put references 30\u201332 before reference 29; because this wiki puts references in order of appearance, order will slightly differ from the original.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181450\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.740 seconds\nReal time usage: 0.758 seconds\nPreprocessor visited node count: 25681\/1000000\nPreprocessor generated node count: 35979\/1000000\nPost\u2010expand include size: 224366\/2097152 bytes\nTemplate argument size: 80421\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 736.853 1 - -total\n 84.62% 623.550 1 - Template:Reflist\n 75.12% 553.503 31 - Template:Citation\/core\n 74.23% 546.940 29 - Template:Cite_journal\n 11.21% 82.568 1 - Template:Infobox_journal_article\n 10.88% 80.166 1 - Template:Infobox\n 9.86% 72.654 58 - Template:Citation\/identifier\n 7.47% 55.051 80 - Template:Infobox\/row\n 4.71% 34.697 2 - Template:Cite_web\n 4.42% 32.541 32 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10119-0!*!0!!en!*!* and timestamp 20181214181449 and revision id 30429\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\">https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","e5903ddcc7734415af1d91fcd258da90_images":[],"e5903ddcc7734415af1d91fcd258da90_timestamp":1544811289,"141c57a5e72031b2191141bec29c26b6_type":"article","141c57a5e72031b2191141bec29c26b6_title":"Global data quality assessment and the situated nature of \u201cbest\u201d research practices in biology (Leonelli 2017)","141c57a5e72031b2191141bec29c26b6_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology","141c57a5e72031b2191141bec29c26b6_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Global data quality assessment and the situated nature of \u201cbest\u201d research practices in biology\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nGlobal data quality assessment and the situated nature of \u201cbest\u201d research practices in biologyJournal\n \nData Science JournalAuthor(s)\n \nLeonelli, SabinaAuthor affiliation(s)\n \nUniversity of Adelaide and University of ExeterPrimary contact\n \nEmail: s dot Leonelli at exeter dot ac dot ukYear published\n \n2017Volume and issue\n \n16Page(s)\n \n32DOI\n \n10.5334\/dsj-2017-032ISSN\n \n1683-1470Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/datascience.codata.org\/articles\/10.5334\/dsj-2017-032\/Download\n \nhttp:\/\/datascience.codata.org\/articles\/10.5334\/dsj-2017-032\/galley\/690\/download\/ (PDF)\n\nContents\n\n1 Abstract \n2 Introduction: Open data and the assessment of data quality in the life sciences \n3 Existing approaches to research data quality assessment \n4 What technology, for which purpose? \n5 Implications for low-resourced research environments \n6 Conclusion: Fostering critical engagement with data quality \n7 Footnotes \n8 Ethics and consent \n9 Acknowledgements \n10 Competing interests \n11 References \n12 Notes \n\n\n\nAbstract \nThis paper reflects on the relation between international debates around data quality assessment and the diversity characterizing research practices, goals and environments within the life sciences. Since the emergence of molecular approaches, many biologists have focused their research, and related methods and instruments for data production, on the study of genes and genomes. While this trend is now shifting, prominent institutions and companies with stakes in molecular biology continue to set standards for what counts as \"good science\" worldwide, resulting in the use of specific data production technologies as proxy for assessing data quality. This is problematic considering (1) the variability in research cultures, goals and the very characteristics of biological systems, which can give rise to countless different approaches to knowledge production; and (2) the existence of research environments that produce high-quality, significant datasets despite not availing themselves of the latest technologies. Ethnographic research carried out in such environments evidences a widespread fear among researchers that providing extensive information about their experimental set-up will affect the perceived quality of their data, making their findings vulnerable to criticisms by better-resourced peers. These fears can make scientists resistant to sharing data or describing their provenance. To counter this, debates around open data need to include critical reflection on how data quality is evaluated, and the extent to which that evaluation requires a localized assessment of the needs, means and goals of each research environment.\nKeywords: data quality, research assessment, peer review, scientific publication, research methods, data generation\n\nIntroduction: Open data and the assessment of data quality in the life sciences \nMuch of the international discussion around open science, and particularly debates around open data, is concerned with how to assess and monitor the quality and reliability of data being disseminated through repositories and databases.[1] Finding reliable ways to guarantee data quality is of great import when attempting to incentivize data sharing and re-use, since trust in the reliability of data available online is crucial to researchers considering them as a starting point for \u2013 or even just complement to \u2013 their ongoing work.[2][3][4]\nIndeed, the quality and reliability of data hosted by digital databases is key to the success of open data, particularly in the wake of the \"replicability crisis\" recently experienced by fields such as psychology and biomedicine[5], and given the constant acceleration of the pace at which researchers produce and publish results.[6] However, the wide variation among the methods, materials, goals, techniques used in pluralistic fields such as biology, as well as the diverse ways in which data can be evaluated depending on the goals of the investigation at hand, make it hard to set common standards and establish international guidelines for evaluating data quality.[1] Attempts to implement peer review of the datasets donated to digital databases are also proving problematic, given the constraints in resources, personnel and expertise experienced by most data infrastructures, and the scarce time and rewards available to researchers contributing expertise to such efforts. This problem is aggravated by the speed with which standards, technologies and knowledge change and develop in any given domain, which makes it difficult, time-intensive and expensive to maintain and update databases and related quality standards as needed.\nThis paper examines the relation between international discussions around how to evaluate data quality, and the existing diversity characterizing research work within the life sciences, particularly in relation to biologists\u2019 access to and use of instruments, infrastructures and materials. Since the molecular bandwagon took off in Europe and the U.S. in the 1950s, the majority of resources and attention within biology has been dedicated to creating methods and technologies to study the lowest levels of organizations of organisms, particularly genomics.[7][8] This trend is now reversing, with substantial interest returning to the ways in which environmental, phenotypic and epigenetic factors interact with molecular components.[9][10][11] However, countries which adopted and supported the molecular approach \u2013 including Japan, China and Singapore \u2013 continue to set the standards for what counts as \"good science\" worldwide. In practice, this means that the technologies and methods fostered by top research sites in these countries \u2013 such as, most glaringly, next generation sequencing methods and instruments \u2013 are often taken as exemplary of best laboratory practice, to the point that the use of software and machines popular in those locations is widely used as proxy for assessing the quality of the resulting findings.\nThis situation turns out to be problematic when considering the sophisticated relationship between the goals and interests of researchers at different locations, the specific characteristics of each target system in biology, and the methods devised to study those systems. These factors may vary and be combined in myriad ways, giving rise to countless different ways to conduct and validate research, and thus to assess the quality of relevant data. It is also troubling when considering research environments that do not have the financial and infrastructural resources to avail themselves of the latest software or instrument, but which are nevertheless producing high-quality data of potential biological significance \u2013 because of the materials they have access to, their innovative conceptual or methodological approach, or their focus on questions and phenomena of little interest to researchers based elsewhere. All too often, researchers working in such environments are afraid that lack of access to the latest technologies will affect the quality and reliability of their data, and will make their findings vulnerable to criticisms by better-resourced peers. These fears can result in researchers being unwilling to share their data and\/or to describe the specific circumstances and tools through which they were obtained, thus making it impossible for others to build on their research and replicate it elsewhere.\nAgainst this background, this paper defends the idea that debates around open data can and should foster critical reflection on how data quality can and should be evaluated, and the extent to which this involves a localized assessment of the challenges, limitations and imperfections characterizing all research environments. To this aim, I first reflect on existing models of data quality assessment in the life sciences and illustrate why the use of specific technologies for data production can end up being deployed as a proxy for data quality. I then discuss the problems with this approach to data quality assessment, focusing both on the history of molecular biology to date and on contemporary perceptions of technological expectations and standards by researchers in both African and European countries. I stress how technologies for data production and dissemination have become markers for researchers\u2019 identity and perception of their own role and status within their fields, in ways that are potentially damaging both to researchers' careers and to scientific advancement as a whole. \nThis discussion is based on observations acquired in the course of ethnographic visits to biological laboratories in Wales, Britain, the United States, Belgium, Germany, Kenya and South Africa; extensive interviews with researchers working on those sites conducted between 2012 and 2016; and discussions on open data and data quality carried out with African members of the Global Young Academy (GYA) as part of my work as coordinator for the open science working group (https:\/\/globalyoungacademy.net\/activities\/open-science\/).[a] I conclude that it is essential for research data to be evaluated in a manner that is localized and context-sensitive, and open data advocates and policies can play a critical role in fostering constructive and inclusive practices of data quality assessment.\n\nExisting approaches to research data quality assessment \nData quality is a notoriously slippery and multifaceted notion, which has long been the subject of scholarly discussion. A comprehensive review of such debates is provided by Luciano Floridi and Phyllis Illari[13], who highlight how the various approaches available, while usefully focusing on aspects such as error detection and countering misinformation, are ultimately tied to domain-specific estimations of what counts as quality and reliability (and for what purposes) that cannot be transferred easily across fields, and sometimes even across specific cases of data use. This does not help towards the development and implementation of mechanisms that can guarantee the quality of the vast amounts of research data stored in large digital repositories for open consultation. Data dissemination through widely available data infrastructures is characteristic of the current open data landscape, which fits the current policy agenda in making research results visible and potentially re-usable by anybody with the skills and interest to explore them. This mode of data dissemination relies on the assumption that the data made accessible online are of sufficient quality to be useful for further investigation. At the same time, data curators and researchers are well-aware that this assumption is problematic and easy to challenge. This is, first, because no data type is \"intrinsically\" trustworthy, but rather data are regarded as reliable on the basis of the methods, instruments, commitments, values and goals employed by the people who generate them[1]; and second, because while it possible to evaluate the quality of data through a review of related metadata, this evaluation typically require expert skills that not all prospective data users possess or care to exercise.[14][b]\nThe problems involved in continuing to develop large research data collections without clear quality benchmarks is widely recognized by academies, institutions and expert bodies involved in open data debates, and debates over data quality feature regularly in meetings of the Research Data Alliance, CODATA and many other learned societies and organizations around the world. While it is impossible to summarize these extensive debates within the scope of this paper, I now briefly examine six modes of data quality evaluation that have been widely employed so far within the sciences, and which continue to hold sway while new solutions are being developed and tested.\nThe first and most common mode of data quality evaluation consists of traditional peer review of research articles where data appear as evidence for scientific claims. The idea here is that whenever scientific publications are refereed, reviewers also need to assess the quality of the data used as evidence for the claims being made, and will not approve of publications grounded on untrustworthy data. Data attached to peer-reviewed publications are therefore often assumed to be of high quality and can be therefore be openly disseminated without problems. However, there are reasons to doubt the effectiveness of this strategy in the current research environment. This only works for data extracted from journal publications, and is of little use when it comes to data that have not yet been analyzed for publication \u2013 thus restricting the scope of databases in ways that many find unacceptable, particularly in the current big data landscape where the velocity with which data are generated has dramatically increased, and a key reason for open dissemination of data is precisely to facilitate their interpretation. It is also not clear that peer review of publications is a reliable way to peer review data. As noted by critics of this approach, traditional peer review focuses on the credibility of methods and claims made in the given publication, not on data per se (which are anyhow often presented within unstructured \"supplementary information\" sections, when they are presented at all[18]). Reviewers are not usually evaluating whether data could usefully be employed to answer research questions other than the one being asked in the paper, and as a result, they provide a skewed evaluation. This could be regarded as an advantage of peer review, since through this system data are always contextualized and assessed in relation to a particular research goal; yet, it does not help to assess the quality of data in contexts of dissemination and re-use. Thus, data curators in charge of retrieving and assessing the quality of data originally published in association with papers need to employ considerable domain-specific expertise to be able to extract the data from existing publications and making them findable and usable. An example of this is the well-known Gene Ontology, whose curators annotate data about gene products by mining published sources and adapting them to common standards and terminology used within the database, which involves considerable labor and skill.[19][20]\nIndeed, a second mode of data quality assessment currently in use relies on evaluations by data curators in charge of data infrastructures. The argument in this case is that these researchers are experts in data dissemination \u2013 they are the data equivalent of a librarian for traditional manuscripts \u2013 and are therefore best equipped to assess whether or not the data considered for online dissemination are trustworthy and of good enough quality for re-use. Hence, in the gene ontology case (cited above), curators not only select which data are of relevance to the categories used in the database, but also assign \"confidence rankings\" to the data depending on what they perceive as the reliability of the source \u2013 a mechanism that certainly assigns considerable responsibility for data quality assessment to those who manage data infrastructures. This solution works reasonably well for relatively small and well-financed data collections, but fails as soon as the funding required to support data curation ceases to exist, or the volume of data becomes so large as to make manual curation impossible. Also, this type of data quality assessment is only as reliable as the curators in charge, especially in cases where data users are too far removed from the development and maintenance of databases to be able or willing to give feedback and check on curators' decisions.\nA third mode of data quality assessment is thus to leave decisions around data quality to those who have generated the data in the first place, which avoids potential misunderstandings between data producers, reviewers and curators. Again, this solution is not ideal. For one thing, existing databases have a hard time getting data producers to post and appropriately annotate their own data (cases such as PomBase, where over half of the authors of relevant papers post and annotate datasets themselves, are far and few between, and typically occur in relatively small and close-knit communities where trust and accountability are high[21]). Furthermore, whatever standards data producers are using to evaluate the quality of their data, it will unavoidably be steeped in the research culture, habits and methods of their own community and subfield, as well as the goals and materials used in their own research. This means that data producers do not typically have the ability to compare different datasets and evaluate their own data in relation to data produced by other research environments, as would be required when assembling a large data infrastructure. Whenever data leave their context of production and enter new contexts of potential re-use, new standards for quality and reliability may well be required, which in turn demands for external assessment and validation from outside the research environment where data were originally generated.\nA fourth method for data quality assessment consists in the employment of automated processes and algorithms, which have the potential to reduce dramatically the manual labor associated with data curation. There is no doubt that automation facilitates a variety of techniques to test the validity, reliability and veracity of data being disseminated, particularly in the context of data linkage facilities and infrastructures.[22] However, such tools typically need to make substantive general assumptions about what types of data are most reliable, which are hard to defend given the user-related nature of data quality metrics and their dependence on the context and goals of data assessment. An interesting model for the development of future data quality assessment processes within the life sciences is provided by the many quality assessment tools used to evaluate clinical data in biomedical research, though that approach relies again on the exercise of human judgement, which in turn results in contentious disparities in its application.\nAs a fifth option, there have been attempts to crowdsource quality assessment by enabling prospective data users to grade the quality of data that they find available on digital databases. While this method holds great promise, it is hard to apply consistently and reliably in a situation where researchers receive little or no credit for engaging with the curation and reuse of existing data sources, and providing feedback to data infrastructures that may enhance their usefulness and long-term sustainability. As a result of the lack of incentive to participate in the curation of open data, most databases operating within the life sciences receive little feedback from their users, despite the (sometimes considerable) effort put into creating channels for users to provide comments and assess the data being disseminated. Moreover, it is perfectly possible that users' judgements differ considerably depending on their research goals and methodological commitments.\nGiven the difficulties encountered by the methods listed above, researchers involved in data quality assessments (for instance, related to data publication or to the inclusion of data into a database) may recur to a sixth, unofficial and implicit method: the reliance on specific technologies for data production as proxy markers for data quality. In this case, specific pieces of equipment, methods and materials are taken to be intrinsically reliable and thus to enhance \u2013 if not guarantee \u2013 the chance that data produced through those techniques and tools will be of good quality. Within the life sciences, prominent examples of such proxies include the use of next generation sequencing machines and mass spectrometry in model organism biology, microbiomes and systems biology; light-producing reporter genes produced by reputable companies in cell and developmental biology; and de novo gene synthesis and design\/simulation software in synthetic biology. These tools are strongly embedded in leading research repertoires within biology and are extensively adopted by laboratories around the world.[23][24] They are typically easy to verify, with well-established protocols in place and little additional expertise or labor needed, giving rise to what philosopher Ulrich Krohs calls \"convenience experimentation.\"[25] And they are typically a good fit for existing open data infrastructures and formats, which are often developed alongside such technologies as part of the same repertoire (as in the case of sequencing data[26]).\n\nWhat technology, for which purpose? \nIt could be argued that researchers in the life sciences have long been dependent on instruments for data classification and interventions on organisms, and that given the crucial role of such tools in knowledge production, reference to the use of technologies as a proxy for data quality is epistemically justified, particularly when this metric is used in conjunction with other evaluation procedures, such as those described above. In this section, I counter this position by pointing out that it takes no account of the powerful market forces at play in the provision and dissemination of (often extremely expensive) research technologies, as well as the distortions that this involves when it comes to evaluating what counts as an ideal research environment \u2013 and thus as \"best practice\" \u2013 in biological research.\nThe power and size of the industrial complex devoted to the development and mass production of research technologies has grown exponentially since the 1950s, in parallel with the growth of the scale and size of biological research worldwide; and with it, the costs, marketing and competition around research tools have spiraled up.[27] The production of lab equipment is now big business particularly in the United States and Europe, with the top 25 companies accounting for 23.6 billion dollars in sales in 2015 alone.[28] This explosion in the market, alongside the priority accorded to technologies that could capture digitally data pertaining to the molecular level of organization of organisms, ended up fueling a perception of sequencing tools and related equipment as an essential part of any biological investigation, whose utilization lends credibility to research results. The monopoly held by the companies Affymetrix and Illumina over the production of genetic assays and microarray data which endured from the mid-1990s to the late 2000s when competitors emerged, is but one example of the way in which competitive marketing has made its way into the best funded labs around the world, and thus into researchers' ideal of what a perfect research setting needs to look like.[29][30] To keep up their revenue, technology providers have a strong incentive as well as the means to set standards for what count as acceptable data in any one area, by pushing the idea that using their tools guarantees high-quality data. The abundant advertisement of lab equipment to be found in any international science journal, including leading publications such as Nature and Science, bears testament to this phenomenon, as do the large spaces allocated to the marketing of research technologies within any respectable international congress in the life sciences. Thus, market forces introduce incentives for biological labs to possess specific pieces of machinery that are not necessarily linked to achieving research excellence, but rather to the desire to be able to use standards and specifications of data formats that are promoted internationally through the marketing of these technologies.\nGiven this situation, it is not surprising that the use of technology as proxy for data quality continues to occur among editorial boards, research institutions and funders, and international research consortia who have the power to determine what counts as \"good\" research practice, including what counts as data quality. This is acknowledged by biologists working in U.K. and U.S. labs that I have interviewed over the last few years. Even in very well-equipped laboratories at established and well-funded research institutions, researchers complained to me about their access to instruments and related materials. Most notably, when interviewed on practices of data production, dissemination and re-use, researchers displayed insecurity and discomfort around the state of their equipment and of their ability to use it. For instance, I encountered statements of unease around:\n\n instruments and materials that their lab did not possess and which the researcher in question did not view as essential to her research, but whose use was requested as ulterior confirmation of her findings by the reviewers of the journals in which she had tried to publish;\n the extent to which the use of the equipment at hand was being maximized for the benefit of research. For example, many UK-based research groups interviewed over their use of high-throughput technologies for data production expressed worries around the level of technical skill required to use those tools, the proficiency with which lab members were operating the technology, and whether their lab was making the most of such tools;\n the extent to which possessing a given piece of equipment may constitute a competitive disadvantage, but forcing researchers to choose specific research directions in order to make sure that the investment made in the machines is justified. This trend is most evident and best documented in the case of genomic sequencing, a technology whose development required a high level of investment by governmental agencies \u2013 an investment on which funders expect to see returns, thus pushing researchers to capitalize on the resulting genomic data;\n the fast-moving technological developments in the relevant field, which makes even very well-established and visible research groups fearful of being left behind or unaware of the latest instruments and techniques on offer.[31]\nSuch widespread insecurity and fears in relation to research environments in the life sciences is not surprising, given the variety of equipment on offer, the high level of technical skill required to use it, the high costs involved in assembling and maintaining an internationally recognized research lab, and the constantly evolving market. Even within well-resourced labs based in prominent and rich institutions, researchers rarely have access to all the technology that they view as potentially relevant to their various projects; and worries around being \"locked-in\" to a given technology, and\/or unable to use it in the most fruitful way, are widespread across highly provisioned research environments. Such worries have arguably grown in parallel to the increased emphasis on transparency and accountability recommended by open science guidelines, and the related explosion of replication experiments pointing to the irreproducibility of many supposedly well-established results. These developments have an enormous potential to improve scientific methods and communication strategies, by eliciting a healthy and necessary preoccupation with producing high-quality, well-justified, intelligible and re-usable results. At the same time, it is important to recognize that open science guidelines and applicability requirements also undermine the implicit trust among peers that so far characterized many areas of biological inquiry, with several researchers confiding to me that they fear being found wanting by colleagues and worry constantly about whether their laboratory set-up and related skills will be recognized as sufficient and well-suited to their line of inquiry.\n\nImplications for low-resourced research environments \nWithin high-resourced research environments, there are many mechanisms in place to mitigate the potentially harmful implications of this breakdown in trust, and to turn open science requirements into an opportunity to develop common standards of best practice. First, researchers working in well-funded labs have the means and opportunity to constantly exchange personnel, visits and equipment (and related reagents and materials) with each other, so as to learn from each other and work collaboratively to maintain quality standards in their field. Secondly, researchers based in internationally visible and powerful institutions are in a good position to propose specific (uses of) technology as gold standard for their peers and have the resources to adapt quickly to emerging repertoires, instruments and trends. Furthermore, such researchers typically have access to at least some well-recognized equipment, which they can make accessible to staff from other labs in exchange to access to other tools.\nThese strategies do not always work in the context of an increasingly diverse and globalized research workforce, and particularly not in research locations which are not easily reachable because of their geographical location, and\/or where there are stark inequalities in access to technologies, related infrastructures and materials, and internationally visible and acknowledged collaborative networks. Many biologists are based in contexts where access to the latest and most expensive technology is not guaranteed, financially viable or even relevant \u2013 for instance, because research focuses on areas such as morphology, physiology, developmental biology, botany, immunology and ethology, where access to the most recent genome sequencer may not matter since the production of molecular data may not be the focus of inquiry. Whether or not it affects research practice and outcomes, lack of access to the latest equipment can make researchers insecure on several fronts, including: what they do not have access to, and how important it may be for their work and\/or adherence to international expectations; technical skills that they may lack; and the very reliability and quality of their data, regardless of whether that depends on having the latest equipment. These are similar fears and insecurities to those experienced by researchers working in high-resourced environments. And yet, researchers in low-resourced environments often do not have access to the kinds of buffer available to their better-equipped colleagues, with severe consequences for their publication strategies. In interviews conducted with researchers in South Africa and Kenya in 2014, for instance, it was clear that insecurity around data production methods and access to technology has a strong impact on researchers\u2019 self-confidence and wish to have visibility, share data and publish work internationally.[12][32][33]\nSuch findings are not unique nor should they be particularly surprising: scholars in science and technology studies and anthropology have long stressed the role of technology as a marker for identity politics, particularly in the African continent.[34] As starkly illustrated recently by work such as Damien Droney's in Ghana[35], Julie Livingston's in Botswana[36], Joanna Crane's in Uganda[37] and Abena Dove Osseo-Asare's across West and East Africa[38], popular culture associates being a scientist with owning spectacular equipment, and this perception filters down to researchers themselves. Equipment is the most visible and concrete marker of wealth in a lab, and it is often interpreted as a signal of the extent to which a research environment in a low-income country can aspire to produce research comparable in quality and significance to that produced by a high-resourced lab. Technology thus becomes a marker for inclusion and a symbol of being part of the Western world in some way \u2013 taking distance from the identity of \"African scientist\" which many researchers find cumbersome and problematic in their dealings with international publishing outlets, funders and institutions. This contributes to the already unequal championing of home-grown scientific approaches and techniques vis-\u00e0-vis methods, concepts and questions imported from the Global North, despite the existence of research areas that are less dependent on expensive machinery and more on elements commonly found across low-resourced environments, such as manpower, expertise and access to specific locations or natural resources.[39]\nThese considerations, which of course apply more widely than African science and potentially include all research conducted in low-resourced environments, bring me to the conclusion that using references to specific technology as proxies for data quality has at least four problematic implications:\n\n1. It may act as an incentive to reduce diversity and creativity in research approaches, by encouraging standardization and the use of the same techniques and technologies regardless of the research context.\nThis situation is troublesome when considering the sophisticated relationship between the goals and interests of researchers at different locations, the specific characteristics of each target system in biology, and the methods devised to study those systems; factors which vary widely and can be combined in myriads of ways, giving rise to countless different ways to conduct and validate research. It is also problematic when considering research environments that cannot avail themselves of the latest software or instrument but which are nevertheless producing high-quality data of potential biological significance \u2013 sometimes because of the materials they have access to, sometimes because of their innovative conceptual or methodological approach, and sometimes because they are targeting questions and phenomena of little interest to researchers based elsewhere.\n\n2. It leads to widespread mistrust and fear of openness, particularly when it comes to the sharing of research data.\nAll too often, researchers working in low-resourced environments are afraid that lack of access to the latest technologies will affect the quality and reliability of their data and will make their findings vulnerable to criticisms by better-resourced peers. Disparity in access to technologies also affects the speed and efficiency with which data being shared are analyzed, giving researchers based in well-equipped labs the opportunity to analyze and publish on data produced in low-resourced conditions much faster than the original data producers (under the current evaluation regimes, which privilege publication of papers over data production, this is equivalent to being scooped). These fears can result in researchers being unwilling to share their data and\/or to describe the specific circumstances and tools through which they were obtained.\n\n3. It reinforces systematic disadvantage among labs that do not have access to expensive resources.\nThis may be the result of researchers' own reluctance to acquire international partners who could question their methods, and\/or to disclose their set-ups (as in the previous point). It may also arise due to the insidious power that assumptions around what counts as a good research environment have within academic structures, evaluation panels and editorial boards of international journals. It is no secret that researchers located in highly reputable institutions have less trouble having their papers accepted for peer review at top-level journals such as Nature and Science. Similarly, many national policies explicitly ask researchers to emulate the working practices of what are typically regarded as scientific leaders at top Western institutions.\n\n4. It encourages misunderstandings and miscommunication between research data producers and users.\nPeople who do not articulate the differences between their environments, or feel compelled to minimize them in the name of implicit good standards for \"best practice,\" are at risk of miscommunications and misunderstandings, leading to breakdown in collaborations, problems in interpreting results and difficulties in replicating experiments.\n\nConclusion: Fostering critical engagement with data quality \nIn this paper, I pointed to data quality assessment as crucial to international research collaboration and advancements in the age of open data. At the same time, I warned that the push towards open data, which involves an increasing emphasis on standard data formats and tools for data sharing, is affected by the extensive commercialization of lab equipment and technologies for data dissemination. These elements risk to create a situation where data quality is assessed on the basis of the technologies being employed, rather than the fit between data, methods, materials available and research questions being asked. By contrast, research strategies are typically fine-tuned to the specific questions that researchers wish to pursue and to the phenomena that they wish to study, and such fine-tuning is conducive to research outputs that are credible, well-justified and innovative in their approach and significance. There is thus a wide variety of models for what may count as \"best practice,\" \"adequate data stewardship\" and \"good research environments,\" whose relevance depends on the specific situations of inquiry in which researchers operate. A molecular biology lab with the latest equipment based at Harvard or Cambridge needs not be the standard against which all research set-ups around the world are set, and it should certainly not be implicitly taken to play that role. What type of experimental set-up fits which research project is a contextual matter, depending on many factors including the research questions and approach that is taken, the expertise of the researchers in question, the social dynamics within the group and its international collaborations, and the institutional support, infrastructures and materials available to researchers.\nThis does not mean that researchers working under very different conditions should not talk with each other and exchange tips for improving their environment and working habits. Quite the contrary: acknowledging diversity is an important step towards making such conversation more meaningful and fruitful, as long as it involves challenging the presumption (often unjustified, as the research above demonstrates) that researchers working within the same field actually mean the same when using similar terminologies, and should be constrained in the same ways regardless of the specificities of their working environment.\nIt is imperative that researchers, policy-makers and funders engaged in debates around data quality take these dimensions into account, particularly when thinking about implementing open data practices in low-resourced research environments. The sharing of data typically relies on the ability to use sophisticated data formats and digital data infrastructures, and thus to keep up with the fast pace of technological change associated to such data sharing tools and standards. This becomes problematic given the importance of storing and disseminating multiple data formats, non-digital sources (as in \"old-fashioned\" paper archives) and data produced by different versions of the same software, which helps to embrace the variety of work carried out in the sciences globally (including both low-resourced laboratories and the so-called \"long tail of science\"). Also, it is crucial to enable researchers to develop their projects whether or not they avail themselves of the latest technology, and hence to consider and assess when such technology is needed, and for which purposes.\nThus, open data initiatives should be aware of the implications of endorsing specific types of technologies (whether hardware, software or specific laboratory instruments) as markers of research quality. Debates around open data should include explicit and field-specific reflections around the relation between data, research instruments and methods, where researchers clearly articulate their assumptions on what constitutes \"best practice,\" who sets a model for such work, and whether such assumptions are realistic and warranted in light of their own research experiences. This type of articulation is a precious tool for research advancement, since it would encourage confrontation and dialogue at the international level around what quality standards are desirable for data, and with respect to which uses and research goals. These reflexive exercises could be of great value in an ever-globalized and diverse scientific landscape, where the specificity of locations, methods and interests characterizing each research community needs to be documented as essential metadata. In the absence of such critical engagement, open data guidelines risk to dismiss or obscure researchers' situated knowledge and practices (as well as the diversity of fundamental research carried out around the world[40]), and instead appeal to politically charged and potentially damaging assumptions about what constitutes \"best practice.\"\n\nFootnotes \n\n\n\u2191 The empirical research for this paper was carried out by me within research sites in Wales, Britain, the United States, Germany and Belgium, and by Louise Bezuidenhout within sites in South Africa and Kenya (for more details on the latter research and related methods, see the paper by Bezuidenhout in this special issue). Given the sensitive nature of the interview materials, the raw data underpinning this paper cannot be openly disseminated; however, a digested and anonymized version of the data is provided on Figshare.[12] \n\n\u2191 It has also been argued that data quality does not matter within big data collections, because existing data can be triangulated with other datasets documenting the same phenomenon, and datasets that corroborate each other can justifiably be viewed as more reliable.[15] Against this view, myself and others pointed out that triangulation only works when there are enough datasets that document the same phenomenon from different angles, which is not always the case in scientific research.[16][17] \n\n\nEthics and consent \nThe interviews and ethnographies used as empirical source in this paper have received approval by the Social Science Ethics Committee of the University of Exeter, in relation to the projects \"Beyond the Digital Divide\" and \"The Epistemology of Data-Intensive Science.\" All participants signed a consent form. Their contributions are anonymized and their confidentiality is fully respected.\n\nAcknowledgements \nThis research was funded by the European Research Council grant award 335925 (\"The Epistemology of Data Science\"), the Leverhulme Trust Grant number RPG-2013-153 (\"Beyond the Digital Divide\"), and the Australian Research Council, Discovery Project DP160102989 (\"Organisms and Us\"). The author gratefully acknowledges input from the Data Studies group at Exeter, and particularly Louise Bezuidenhout, Brian Rappert and Ann Kelly as co-team members of the Leverhulme Project; participants to the \"Pacing Science\" workshop that took place at the University of Exeter in May 2016, where this paper was first presented, and particularly Linsey McGoey and Simon Hodson; members of the Global Young Academy \"Open Science\" and \"Global Access to Research Software\" working groups, especially Abdullah Shams Bin Tariq and Martin Dominik; members of the Knowledge\/Value research network, particularly Kaushik Sunder Rajan and Kris Peterson; and the dozens of researchers who spared time and effort to discuss their methods, outputs and working conditions with me and my colleagues.\n\nCompeting interests \nThe author has no competing interests to declare.\n\nReferences \n\n\n\u2191 1.0 1.1 1.2 Cai, L.; Zhu, Y. (2015). \"The challenges of data quality and data quality assessment in the big data era\". Data Science Journal 14: 2. doi:10.5334\/dsj-2015-002.   \n\n\u2191 Ossorio, P.N. (2011). \"Bodies of data: Genomic data and bioscience data sharing\". Social Research 78 (3): 907-932. PMC PMC3984581. PMID 24733955. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3984581 .   \n\n\u2191 Borgman, C.L. (2012). \"The conundrum of sharing research data\". Journal for the Association for Information Science and Technology 63 (6): 1059\u20131078. doi:10.1002\/asi.22634.   \n\n\u2191 Leonelli, S. (2016). Data-Centric Biology: A Philosophical Study. University of Chicago Press. pp. 288. ISBN 9780226416472. http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/D\/bo24957334.html .   \n\n\u2191 Allison, D.B.; Brown, A.W.; George, B.J.; Kaiser, K.A.. \"Reproducibility: A tragedy of errors\". Nature 530 (7588): 27\u20139. doi:10.1038\/530027a. PMC PMC4831566. PMID 26842041. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4831566 .   \n\n\u2191 Pulverer, B.. \"Reproducibility blues\". EMBO Journal 34 (22): 2721-4. doi:10.15252\/embj.201570090. PMC PMC4682652. PMID 26538323. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4682652 .   \n\n\u2191 Nowotny, H.; Testa, G. (2011). Naked Genes. The MIT Press. pp. 152. ISBN 9780262014939. https:\/\/mitpress.mit.edu\/books\/naked-genes .   \n\n\u2191 M\u00fcller-Wille, S.W.; Rheinberger, H.J. (2012). A Cultural History of Heredity. University of Chicago Press. pp. 288. ISBN 9780226545721. http:\/\/www.press.uchicago.edu\/ucp\/books\/book\/chicago\/C\/bo8787518.html .   \n\n\u2191 Barnes, B.; Dupr\u00e9, J. (2009). Genomes and What to Make of Them. University of Chicago Press. pp. 288. ISBN 9780226172965. http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/G\/bo5705879.html .   \n\n\u2191 Dupr\u00e9, J. (2012). Processes of Life. Oxford University Press. pp. 320. ISBN 9780199691982. https:\/\/global.oup.com\/academic\/product\/processes-of-life-9780199691982?cc=us&lang=en& .   \n\n\u2191 M\u00fcller-Wille, S.W.; Rheinberger, H.J. (2017). The Gene: From Genetics to Postgenomics. University of Chicago Press. pp. 176. ISBN 9780226510002. http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/G\/bo20952390.html .   \n\n\u2191 12.0 12.1 Bezuidenhout, L.; Rappert, B.; Leonelli, S.; Kelly, A.H. (2016). \"Beyond the Digital Divide: Sharing Research Data across Developing and Developed Countries\". figshare. https:\/\/figshare.com\/articles\/Beyond_the_Digital_Divide_Sharing_Research_Data_across_Developing_and_Developed_Countries\/3203809\/1 .   \n\n\u2191 Floridi, L.; Illari, P. (2014). The Philosophy of Information Quality. Springer International Publishing. pp. 315. doi:10.1007\/978-3-319-07121-3. ISBN 9783319071213.   \n\n\u2191 Leonelli, S.. \"Locating ethics in data science: Responsibility and accountability in global and distributed knowledge production systems\". Philosophical Transactions of the Royal Society A 374 (2083): 20160122. doi:10.1098\/rsta.2016.0122. PMC PMC5124067. PMID 28336799. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5124067 .   \n\n\u2191 Mayer-Sch\u00f6nberger, V., Cukier, K. (2013). Big Data: A Revolution That Will Transform How We Live, Work and Think. John Murray. pp. 256. ISBN 9781848547926.   \n\n\u2191 Leonelli, S.. \"What Difference Does Quantity Make? On the Epistemology of Big Data in Biology\". Big Data and Society 1 (1). doi:10.1177\/2053951714534395. PMC PMC4340542. PMID 25729586. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4340542 .   \n\n\u2191 Calude, C.S.; Longo, G.. \"The Deluge of Spurious Correlations in Big Data\". Foundations of Science 21: 1\u201318. doi:10.1007\/s10699-016-9489-4.   \n\n\u2191 Morey, R.D.; Chambers, C.D.; Etchells, P.J. et al.. \"The Peer Reviewers' Openness Initiative: Incentivizing open research practices through peer review\". Royal Society Open Science 3 (1): 150547. doi:10.1098\/rsos.150547. PMC PMC4736937. PMID 26909182. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4736937 .   \n\n\u2191 Leonelli, S.; Diehl, A.D.; Christie, K.R. et al.. \"How the gene ontology evolves\". BMC Bioinformatics 12: 325. doi:10.1186\/1471-2105-12-325. PMC PMC3166943. PMID 21819553. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3166943 .   \n\n\u2191 Blake, J.A.; Christie, K.R.; Dolan, M.E. et al.. \"Gene Ontology Consortium: Going forward\". Nucleic Acids Research 43 (DB1): D1049-56. doi:10.1093\/nar\/gku1179. PMC PMC4383973. PMID 25428369. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383973 .   \n\n\u2191 McDowall, M.D.; Harris, M.A.; Lock, A. et al.. \"PomBase 2015: Updates to the fission yeast database\". Nucleic Acids Research 43 (DB1): D656-61. doi:10.1093\/nar\/gku1040. PMC PMC4383888. PMID 25361970. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383888 .   \n\n\u2191 Kambatla, K.; Kollias, G.; Kumar, V.; Grama, A.. \"Trends in big data analytics\". Journal of Parallel and Distributed Computing 74 (7): 2561-2573. doi:10.1016\/j.jpdc.2014.01.003.   \n\n\u2191 Ankeny, R.A.; Leonelli, S. (2015). \"Valuing Data in Postgenomic Biology: How Data Donation and Curation Practices Challenge the Scientific Publication System\". Postgenomics: Perspectives on Biology after the Genome. Duke University Press. pp. 126\u2013149. ISBN 9780822358947. https:\/\/www.dukeupress.edu\/postgenomics .   \n\n\u2191 Ankeny, R.A.; Leonelli, S.. \"Repertoires: A post-Kuhnian perspective on scientific change and collaborative research\". Studies in History and Philosophy of Science 60: 18\u201328. doi:10.1016\/j.shpsa.2016.08.003. PMID 27938718.   \n\n\u2191 Krohs, U.. \"Convenience experimentation\". Studies in History and Philosophy of Biological and Biomedical Sciences 43 (1): 52-7. doi:10.1016\/j.shpsc.2011.10.005. PMID 22326072.   \n\n\u2191 Leonelli, S.; Ankeny, R.A.. \"Repertoires: How to Transform a Project into a Research Community\". BioScience 65 (7): 701\u2013708. doi:10.1093\/biosci\/biv061. PMC PMC4580990. PMID 26412866. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4580990 .   \n\n\u2191 Rajan, K.S. (2006). Biocapital: The Constitution of Postgenomic Life. Duke University Press. pp. 360. ISBN 9780822337201. https:\/\/www.dukeupress.edu\/biocapital .   \n\n\u2191 Thayer, A.M. (25 April 2016). \"Top instrument firms in 2015\". C&EN 94 (17): 32\u201335. http:\/\/cen.acs.org\/articles\/94\/i17\/Top-instrument-firms-2015.html .   \n\n\u2191 Rogers, S.; Cambrosio, A. (2007). \"Making a new technology work: The standardization and regulation of microarrays\". Yale Journal of Biology and Medicine 80 (4): 165-78. PMC PMC2347363. PMID 18449388. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2347363 .   \n\n\u2191 Kalorama Information (January 2016). \"The World Market for Microarrays\". Research and Markets. https:\/\/www.researchandmarkets.com\/reports\/3619329\/the-world-market-for-microarrays .   \n\n\u2191 Levin, N.; Leonelli, S. Weckowska, D. et al. (2016). \"How Do Scientists Define Openness? Exploring the Relationship Between Open Science Policies and Research Practice\". Bulletin of Science, Technology, & Society 36 (2): 128-141. doi:10.1177\/0270467616668760. PMC PMC5066505. PMID 27807390. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5066505 .   \n\n\u2191 Bezuidenhout, L.; Leonelli, S.; Kelly, A.H., Rappert, B. (2017). \"Beyond the digital divide: Towards a situated approach to open data\". Science and Public Policy. doi:10.1093\/scipol\/scw036.   \n\n\u2191 Bezuidenhout, L. (2017). \"Technology Transfer and True Transformation: Implications for Open Data\". Data Science Journal 16: 26. doi:10.5334\/dsj-2017-026.   \n\n\u2191 Ferguson, J. (2006). Global Shadows: Africa in the Neoliberal World Order. Duke University Press. doi:10.1215\/9780822387640. ISBN 9780822387640. http:\/\/read.dukeupress.edu\/content\/global-shadows .   \n\n\u2191 Droney, D. (2014). \"Ironies of Laboratory Work during Ghana's Second Age of Optimism\". Cultural Anthropology 29 (2): 363\u2013384. doi:10.14506\/ca29.2.10.   \n\n\u2191 Livingston, J. (2012). Improvising Medicine: An African Oncology Ward in an Emerging Cancer Epidemic. Duke University Press. doi:10.1215\/9780822395768. ISBN 9780822395768. http:\/\/read.dukeupress.edu\/content\/improvising-medicine .   \n\n\u2191 Crane, J.T. (2013). Scrambling for Africa: AIDS, Expertise, and the Rise of American Global Health Science. Cornell University Press. pp. 224. ISBN 9780801451959. http:\/\/www.cornellpress.cornell.edu\/book\/?GCOI=80140100922670 .   \n\n\u2191 Osseo-Asare, A.D. (2014). Bitter Roots: The Search for Healing Plants in Africa. University of Chicago Press. pp. 288. ISBN 9780226086163. http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/B\/bo17031345.html .   \n\n\u2191 Kelly, A.; Lezaun, J. (2017). \"The wild indoors: Room-spaces of scientific inquiry\". Cultural Anthropology. \"In press\"   \n\n\u2191 Rochmyaningsih, D.. \"The developing world needs basic research too\". Nature 534 (7605): 7. doi:10.1038\/534007a. PMID 27251238.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, including regionalizing spelling. In some cases important information was missing from the references, and that information was added. The original article had citations listed alphabetically; they are listed in the order they appear here due to the way the wiki works. In several cases, the original article cited sources inline (e.g., Primiero 2014, Stegenga 2014, and Hilgartner 2017) but failed to list them in the References section, or the article didn't even exist yet (Rappert this issue); these have been omitted here. One URL was broken in the original but updated with the current one in this version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\">https:\/\/www.limswiki.org\/index.php\/Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2017)LIMSwiki journal articles (all)LIMSwiki journal articles on open dataLIMSwiki journal articles on informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 27 June 2017, at 18:11.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 932 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","141c57a5e72031b2191141bec29c26b6_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Global_data_quality_assessment_and_the_situated_nature_of_\u201cbest\u201d_research_practices_in_biology skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Global data quality assessment and the situated nature of \u201cbest\u201d research practices in biology<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>This paper reflects on the relation between international debates around data quality assessment and the diversity characterizing research practices, goals and environments within the life sciences. Since the emergence of molecular approaches, many biologists have focused their research, and related methods and instruments for data production, on the study of genes and genomes. While this trend is now shifting, prominent institutions and companies with stakes in molecular biology continue to set standards for what counts as \"good science\" worldwide, resulting in the use of specific data production technologies as proxy for assessing data quality. This is problematic considering (1) the variability in research cultures, goals and the very characteristics of biological systems, which can give rise to countless different approaches to knowledge production; and (2) the existence of research environments that produce high-quality, significant datasets despite not availing themselves of the latest technologies. Ethnographic research carried out in such environments evidences a widespread fear among researchers that providing extensive information about their experimental set-up will affect the perceived quality of their data, making their findings vulnerable to criticisms by better-resourced peers. These fears can make scientists resistant to sharing data or describing their provenance. To counter this, debates around open data need to include critical reflection on how data quality is evaluated, and the extent to which that evaluation requires a localized assessment of the needs, means and goals of each research environment.\n<\/p><p><b>Keywords<\/b>: data quality, research assessment, peer review, scientific publication, research methods, data generation\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction:_Open_data_and_the_assessment_of_data_quality_in_the_life_sciences\">Introduction: Open data and the assessment of data quality in the life sciences<\/span><\/h2>\n<p>Much of the international discussion around open science, and particularly debates around open data, is concerned with how to assess and monitor the quality and reliability of data being disseminated through repositories and databases.<sup id=\"rdp-ebb-cite_ref-CaiTheChallenges15_1-0\" class=\"reference\"><a href=\"#cite_note-CaiTheChallenges15-1\" rel=\"external_link\">[1]<\/a><\/sup> Finding reliable ways to guarantee data quality is of great import when attempting to incentivize data sharing and re-use, since trust in the reliability of data available online is crucial to researchers considering them as a starting point for \u2013 or even just complement to \u2013 their ongoing work.<sup id=\"rdp-ebb-cite_ref-OssorioBodies11_2-0\" class=\"reference\"><a href=\"#cite_note-OssorioBodies11-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BorgmanTheCon12_3-0\" class=\"reference\"><a href=\"#cite_note-BorgmanTheCon12-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LeonelliData16_4-0\" class=\"reference\"><a href=\"#cite_note-LeonelliData16-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/p><p>Indeed, the quality and reliability of data hosted by digital databases is key to the success of open data, particularly in the wake of the \"replicability crisis\" recently experienced by fields such as psychology and biomedicine<sup id=\"rdp-ebb-cite_ref-AllisonRepro16_5-0\" class=\"reference\"><a href=\"#cite_note-AllisonRepro16-5\" rel=\"external_link\">[5]<\/a><\/sup>, and given the constant acceleration of the pace at which researchers produce and publish results.<sup id=\"rdp-ebb-cite_ref-PulvererRepro15_6-0\" class=\"reference\"><a href=\"#cite_note-PulvererRepro15-6\" rel=\"external_link\">[6]<\/a><\/sup> However, the wide variation among the methods, materials, goals, techniques used in pluralistic fields such as biology, as well as the diverse ways in which data can be evaluated depending on the goals of the investigation at hand, make it hard to set common standards and establish international guidelines for evaluating data quality.<sup id=\"rdp-ebb-cite_ref-CaiTheChallenges15_1-1\" class=\"reference\"><a href=\"#cite_note-CaiTheChallenges15-1\" rel=\"external_link\">[1]<\/a><\/sup> Attempts to implement peer review of the datasets donated to digital databases are also proving problematic, given the constraints in resources, personnel and expertise experienced by most data infrastructures, and the scarce time and rewards available to researchers contributing expertise to such efforts. This problem is aggravated by the speed with which standards, technologies and knowledge change and develop in any given domain, which makes it difficult, time-intensive and expensive to maintain and update databases and related quality standards as needed.\n<\/p><p>This paper examines the relation between international discussions around how to evaluate data quality, and the existing diversity characterizing research work within the life sciences, particularly in relation to biologists\u2019 access to and use of instruments, infrastructures and materials. Since the molecular bandwagon took off in Europe and the U.S. in the 1950s, the majority of resources and attention within biology has been dedicated to creating methods and technologies to study the lowest levels of organizations of organisms, particularly genomics.<sup id=\"rdp-ebb-cite_ref-NowotnyNaked11_7-0\" class=\"reference\"><a href=\"#cite_note-NowotnyNaked11-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-M.C3.BCller-WilleACult12_8-0\" class=\"reference\"><a href=\"#cite_note-M.C3.BCller-WilleACult12-8\" rel=\"external_link\">[8]<\/a><\/sup> This trend is now reversing, with substantial interest returning to the ways in which environmental, phenotypic and epigenetic factors interact with molecular components.<sup id=\"rdp-ebb-cite_ref-BarnesGenomes09_9-0\" class=\"reference\"><a href=\"#cite_note-BarnesGenomes09-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Dupr.C3.A9Processes12_10-0\" class=\"reference\"><a href=\"#cite_note-Dupr.C3.A9Processes12-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-M.C3.BCller-WilleTheGene17_11-0\" class=\"reference\"><a href=\"#cite_note-M.C3.BCller-WilleTheGene17-11\" rel=\"external_link\">[11]<\/a><\/sup> However, countries which adopted and supported the molecular approach \u2013 including Japan, China and Singapore \u2013 continue to set the standards for what counts as \"good science\" worldwide. In practice, this means that the technologies and methods fostered by top research sites in these countries \u2013 such as, most glaringly, next generation sequencing methods and instruments \u2013 are often taken as exemplary of best laboratory practice, to the point that the use of software and machines popular in those locations is widely used as proxy for assessing the quality of the resulting findings.\n<\/p><p>This situation turns out to be problematic when considering the sophisticated relationship between the goals and interests of researchers at different locations, the specific characteristics of each target system in biology, and the methods devised to study those systems. These factors may vary and be combined in myriad ways, giving rise to countless different ways to conduct and validate research, and thus to assess the quality of relevant data. It is also troubling when considering research environments that do not have the financial and infrastructural resources to avail themselves of the latest software or instrument, but which are nevertheless producing high-quality data of potential biological significance \u2013 because of the materials they have access to, their innovative conceptual or methodological approach, or their focus on questions and phenomena of little interest to researchers based elsewhere. All too often, researchers working in such environments are afraid that lack of access to the latest technologies will affect the quality and reliability of their data, and will make their findings vulnerable to criticisms by better-resourced peers. These fears can result in researchers being unwilling to share their data and\/or to describe the specific circumstances and tools through which they were obtained, thus making it impossible for others to build on their research and replicate it elsewhere.\n<\/p><p>Against this background, this paper defends the idea that <i>debates around open data can and should foster critical reflection on how data quality can and should be evaluated, and the extent to which this involves a localized assessment of the challenges, limitations and imperfections characterizing all research environments.<\/i> To this aim, I first reflect on existing models of data quality assessment in the life sciences and illustrate why the use of specific technologies for data production can end up being deployed as a proxy for data quality. I then discuss the problems with this approach to data quality assessment, focusing both on the history of molecular biology to date and on contemporary perceptions of technological expectations and standards by researchers in both African and European countries. I stress how technologies for data production and dissemination have become markers for researchers\u2019 identity and perception of their own role and status within their fields, in ways that are potentially damaging both to researchers' careers and to scientific advancement as a whole. \n<\/p><p>This discussion is based on observations acquired in the course of ethnographic visits to biological laboratories in Wales, Britain, the United States, Belgium, Germany, Kenya and South Africa; extensive interviews with researchers working on those sites conducted between 2012 and 2016; and discussions on open data and data quality carried out with African members of the Global Young Academy (GYA) as part of my work as coordinator for the open science working group (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/globalyoungacademy.net\/activities\/open-science\/\" target=\"_blank\">https:\/\/globalyoungacademy.net\/activities\/open-science\/<\/a>).<sup id=\"rdp-ebb-cite_ref-13\" class=\"reference\"><a href=\"#cite_note-13\" rel=\"external_link\">[a]<\/a><\/sup> I conclude that it is essential for research data to be evaluated in a manner that is localized and context-sensitive, and open data advocates and policies can play a critical role in fostering constructive and inclusive practices of data quality assessment.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Existing_approaches_to_research_data_quality_assessment\">Existing approaches to research data quality assessment<\/span><\/h2>\n<p>Data quality is a notoriously slippery and multifaceted notion, which has long been the subject of scholarly discussion. A comprehensive review of such debates is provided by Luciano Floridi and Phyllis Illari<sup id=\"rdp-ebb-cite_ref-FloridiThePhil14_14-0\" class=\"reference\"><a href=\"#cite_note-FloridiThePhil14-14\" rel=\"external_link\">[13]<\/a><\/sup>, who highlight how the various approaches available, while usefully focusing on aspects such as error detection and countering misinformation, are ultimately tied to domain-specific estimations of what counts as quality and reliability (and for what purposes) that cannot be transferred easily across fields, and sometimes even across specific cases of data use. This does not help towards the development and implementation of mechanisms that can guarantee the quality of the vast amounts of research data stored in large digital repositories for open consultation. Data dissemination through widely available data infrastructures is characteristic of the current open data landscape, which fits the current policy agenda in making research results visible and potentially re-usable by anybody with the skills and interest to explore them. This mode of data dissemination relies on the assumption that the data made accessible online are of sufficient quality to be useful for further investigation. At the same time, data curators and researchers are well-aware that this assumption is problematic and easy to challenge. This is, first, because no data type is \"intrinsically\" trustworthy, but rather data are regarded as reliable on the basis of the methods, instruments, commitments, values and goals employed by the people who generate them<sup id=\"rdp-ebb-cite_ref-CaiTheChallenges15_1-2\" class=\"reference\"><a href=\"#cite_note-CaiTheChallenges15-1\" rel=\"external_link\">[1]<\/a><\/sup>; and second, because while it possible to evaluate the quality of data through a review of related metadata, this evaluation typically require expert skills that not all prospective data users possess or care to exercise.<sup id=\"rdp-ebb-cite_ref-LeonelliLocating16_15-0\" class=\"reference\"><a href=\"#cite_note-LeonelliLocating16-15\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-19\" class=\"reference\"><a href=\"#cite_note-19\" rel=\"external_link\">[b]<\/a><\/sup>\n<\/p><p>The problems involved in continuing to develop large research data collections without clear quality benchmarks is widely recognized by academies, institutions and expert bodies involved in open data debates, and debates over data quality feature regularly in meetings of the Research Data Alliance, CODATA and many other learned societies and organizations around the world. While it is impossible to summarize these extensive debates within the scope of this paper, I now briefly examine six modes of data quality evaluation that have been widely employed so far within the sciences, and which continue to hold sway while new solutions are being developed and tested.\n<\/p><p>The first and most common mode of data quality evaluation consists of <i>traditional peer review<\/i> of research articles where data appear as evidence for scientific claims. The idea here is that whenever scientific publications are refereed, reviewers also need to assess the quality of the data used as evidence for the claims being made, and will not approve of publications grounded on untrustworthy data. Data attached to peer-reviewed publications are therefore often assumed to be of high quality and can be therefore be openly disseminated without problems. However, there are reasons to doubt the effectiveness of this strategy in the current research environment. This only works for data extracted from journal publications, and is of little use when it comes to data that have not yet been analyzed for publication \u2013 thus restricting the scope of databases in ways that many find unacceptable, particularly in the current big data landscape where the velocity with which data are generated has dramatically increased, and a key reason for open dissemination of data is precisely to facilitate their interpretation. It is also not clear that peer review of publications is a reliable way to peer review data. As noted by critics of this approach, traditional peer review focuses on the credibility of methods and claims made in the given publication, not on data per se (which are anyhow often presented within unstructured \"supplementary information\" sections, when they are presented at all<sup id=\"rdp-ebb-cite_ref-MoreyThePeer16_20-0\" class=\"reference\"><a href=\"#cite_note-MoreyThePeer16-20\" rel=\"external_link\">[18]<\/a><\/sup>). Reviewers are not usually evaluating whether data could usefully be employed to answer research questions other than the one being asked in the paper, and as a result, they provide a skewed evaluation. This could be regarded as an advantage of peer review, since through this system data are always contextualized and assessed in relation to a particular research goal; yet, it does not help to assess the quality of data in contexts of dissemination and re-use. Thus, data curators in charge of retrieving and assessing the quality of data originally published in association with papers need to employ considerable domain-specific expertise to be able to extract the data from existing publications and making them findable and usable. An example of this is the well-known Gene Ontology, whose curators annotate data about gene products by mining published sources and adapting them to common standards and terminology used within the database, which involves considerable labor and skill.<sup id=\"rdp-ebb-cite_ref-LeonelliHowTheGene11_21-0\" class=\"reference\"><a href=\"#cite_note-LeonelliHowTheGene11-21\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BlakeGeneOnt15_22-0\" class=\"reference\"><a href=\"#cite_note-BlakeGeneOnt15-22\" rel=\"external_link\">[20]<\/a><\/sup>\n<\/p><p>Indeed, a second mode of data quality assessment currently in use relies on <i>evaluations by data curators in charge of data infrastructures<\/i>. The argument in this case is that these researchers are experts in data dissemination \u2013 they are the data equivalent of a librarian for traditional manuscripts \u2013 and are therefore best equipped to assess whether or not the data considered for online dissemination are trustworthy and of good enough quality for re-use. Hence, in the gene ontology case (cited above), curators not only select which data are of relevance to the categories used in the database, but also assign \"confidence rankings\" to the data depending on what they perceive as the reliability of the source \u2013 a mechanism that certainly assigns considerable responsibility for data quality assessment to those who manage data infrastructures. This solution works reasonably well for relatively small and well-financed data collections, but fails as soon as the funding required to support data curation ceases to exist, or the volume of data becomes so large as to make manual curation impossible. Also, this type of data quality assessment is only as reliable as the curators in charge, especially in cases where data users are too far removed from the development and maintenance of databases to be able or willing to give feedback and check on curators' decisions.\n<\/p><p>A third mode of data quality assessment is thus <i>to leave decisions around data quality to those who have generated the data in the first place<\/i>, which avoids potential misunderstandings between data producers, reviewers and curators. Again, this solution is not ideal. For one thing, existing databases have a hard time getting data producers to post and appropriately annotate their own data (cases such as PomBase, where over half of the authors of relevant papers post and annotate datasets themselves, are far and few between, and typically occur in relatively small and close-knit communities where trust and accountability are high<sup id=\"rdp-ebb-cite_ref-McDowallPomBase15_23-0\" class=\"reference\"><a href=\"#cite_note-McDowallPomBase15-23\" rel=\"external_link\">[21]<\/a><\/sup>). Furthermore, whatever standards data producers are using to evaluate the quality of their data, it will unavoidably be steeped in the research culture, habits and methods of their own community and subfield, as well as the goals and materials used in their own research. This means that data producers do not typically have the ability to compare different datasets and evaluate their own data in relation to data produced by other research environments, as would be required when assembling a large data infrastructure. Whenever data leave their context of production and enter new contexts of potential re-use, new standards for quality and reliability may well be required, which in turn demands for external assessment and validation from outside the research environment where data were originally generated.\n<\/p><p>A fourth method for data quality assessment consists in <i>the employment of automated processes and algorithms<\/i>, which have the potential to reduce dramatically the manual labor associated with data curation. There is no doubt that automation facilitates a variety of techniques to test the validity, reliability and veracity of data being disseminated, particularly in the context of data linkage facilities and infrastructures.<sup id=\"rdp-ebb-cite_ref-KambatlaTrends14_24-0\" class=\"reference\"><a href=\"#cite_note-KambatlaTrends14-24\" rel=\"external_link\">[22]<\/a><\/sup> However, such tools typically need to make substantive general assumptions about what types of data are most reliable, which are hard to defend given the user-related nature of data quality metrics and their dependence on the context and goals of data assessment. An interesting model for the development of future data quality assessment processes within the life sciences is provided by the many quality assessment tools used to evaluate clinical data in biomedical research, though that approach relies again on the exercise of human judgement, which in turn results in contentious disparities in its application.\n<\/p><p>As a fifth option, there have been attempts <i>to crowdsource quality assessment<\/i> by enabling prospective data users to grade the quality of data that they find available on digital databases. While this method holds great promise, it is hard to apply consistently and reliably in a situation where researchers receive little or no credit for engaging with the curation and reuse of existing data sources, and providing feedback to data infrastructures that may enhance their usefulness and long-term sustainability. As a result of the lack of incentive to participate in the curation of open data, most databases operating within the life sciences receive little feedback from their users, despite the (sometimes considerable) effort put into creating channels for users to provide comments and assess the data being disseminated. Moreover, it is perfectly possible that users' judgements differ considerably depending on their research goals and methodological commitments.\n<\/p><p>Given the difficulties encountered by the methods listed above, researchers involved in data quality assessments (for instance, related to data publication or to the inclusion of data into a database) may recur to a sixth, unofficial and implicit method: the <i>reliance on specific technologies for data production as proxy markers for data quality<\/i>. In this case, specific pieces of equipment, methods and materials are taken to be intrinsically reliable and thus to enhance \u2013 if not guarantee \u2013 the chance that data produced through those techniques and tools will be of good quality. Within the life sciences, prominent examples of such proxies include the use of next generation sequencing machines and mass spectrometry in model organism biology, microbiomes and systems biology; light-producing reporter genes produced by reputable companies in cell and developmental biology; and <i>de novo<\/i> gene synthesis and design\/simulation software in synthetic biology. These tools are strongly embedded in leading research repertoires within biology and are extensively adopted by laboratories around the world.<sup id=\"rdp-ebb-cite_ref-AnkenyValuing15_25-0\" class=\"reference\"><a href=\"#cite_note-AnkenyValuing15-25\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AnkenyRepertoires16_26-0\" class=\"reference\"><a href=\"#cite_note-AnkenyRepertoires16-26\" rel=\"external_link\">[24]<\/a><\/sup> They are typically easy to verify, with well-established protocols in place and little additional expertise or labor needed, giving rise to what philosopher Ulrich Krohs calls \"convenience experimentation.\"<sup id=\"rdp-ebb-cite_ref-KrohsConvenience12_27-0\" class=\"reference\"><a href=\"#cite_note-KrohsConvenience12-27\" rel=\"external_link\">[25]<\/a><\/sup> And they are typically a good fit for existing open data infrastructures and formats, which are often developed alongside such technologies as part of the same repertoire (as in the case of sequencing data<sup id=\"rdp-ebb-cite_ref-LeonelliRepertoires15_28-0\" class=\"reference\"><a href=\"#cite_note-LeonelliRepertoires15-28\" rel=\"external_link\">[26]<\/a><\/sup>).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"What_technology.2C_for_which_purpose.3F\">What technology, for which purpose?<\/span><\/h2>\n<p>It could be argued that researchers in the life sciences have long been dependent on instruments for data classification and interventions on organisms, and that given the crucial role of such tools in knowledge production, reference to the use of technologies as a proxy for data quality is epistemically justified, particularly when this metric is used in conjunction with other evaluation procedures, such as those described above. In this section, I counter this position by pointing out that it takes no account of the powerful market forces at play in the provision and dissemination of (often extremely expensive) research technologies, as well as the distortions that this involves when it comes to evaluating what counts as an ideal research environment \u2013 and thus as \"best practice\" \u2013 in biological research.\n<\/p><p>The power and size of the industrial complex devoted to the development and mass production of research technologies has grown exponentially since the 1950s, in parallel with the growth of the scale and size of biological research worldwide; and with it, the costs, marketing and competition around research tools have spiraled up.<sup id=\"rdp-ebb-cite_ref-RajanBiocapital06_29-0\" class=\"reference\"><a href=\"#cite_note-RajanBiocapital06-29\" rel=\"external_link\">[27]<\/a><\/sup> The production of lab equipment is now big business particularly in the United States and Europe, with the top 25 companies accounting for 23.6 billion dollars in sales in 2015 alone.<sup id=\"rdp-ebb-cite_ref-ThayerTopInst16_30-0\" class=\"reference\"><a href=\"#cite_note-ThayerTopInst16-30\" rel=\"external_link\">[28]<\/a><\/sup> This explosion in the market, alongside the priority accorded to technologies that could capture digitally data pertaining to the molecular level of organization of organisms, ended up fueling a perception of sequencing tools and related equipment as an essential part of any biological investigation, whose utilization lends credibility to research results. The monopoly held by the companies Affymetrix and Illumina over the production of genetic assays and microarray data which endured from the mid-1990s to the late 2000s when competitors emerged, is but one example of the way in which competitive marketing has made its way into the best funded labs around the world, and thus into researchers' ideal of what a perfect research setting needs to look like.<sup id=\"rdp-ebb-cite_ref-RogersMaking07_31-0\" class=\"reference\"><a href=\"#cite_note-RogersMaking07-31\" rel=\"external_link\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KaloramaTheWorld16_32-0\" class=\"reference\"><a href=\"#cite_note-KaloramaTheWorld16-32\" rel=\"external_link\">[30]<\/a><\/sup> To keep up their revenue, technology providers have a strong incentive as well as the means to set standards for what count as acceptable data in any one area, by pushing the idea that using their tools guarantees high-quality data. The abundant advertisement of lab equipment to be found in any international science journal, including leading publications such as <i>Nature<\/i> and <i>Science<\/i>, bears testament to this phenomenon, as do the large spaces allocated to the marketing of research technologies within any respectable international congress in the life sciences. Thus, market forces introduce incentives for biological labs to possess specific pieces of machinery that are not necessarily linked to achieving research excellence, but rather to the desire to be able to use standards and specifications of data formats that are promoted internationally through the marketing of these technologies.\n<\/p><p>Given this situation, it is not surprising that the use of technology as proxy for data quality continues to occur among editorial boards, research institutions and funders, and international research consortia who have the power to determine what counts as \"good\" research practice, including what counts as data quality. This is acknowledged by biologists working in U.K. and U.S. labs that I have interviewed over the last few years. Even in very well-equipped laboratories at established and well-funded research institutions, researchers complained to me about their access to instruments and related materials. Most notably, when interviewed on practices of data production, dissemination and re-use, researchers displayed insecurity and discomfort around the state of their equipment and of their ability to use it. For instance, I encountered statements of unease around:\n<\/p>\n<ul><li> instruments and materials that their lab did not possess and which the researcher in question did not view as essential to her research, but whose use was requested as ulterior confirmation of her findings by the reviewers of the journals in which she had tried to publish;<\/li><\/ul>\n<ul><li> the extent to which the use of the equipment at hand was being maximized for the benefit of research. For example, many UK-based research groups interviewed over their use of high-throughput technologies for data production expressed worries around the level of technical skill required to use those tools, the proficiency with which lab members were operating the technology, and whether their lab was making the most of such tools;<\/li><\/ul>\n<ul><li> the extent to which possessing a given piece of equipment may constitute a competitive disadvantage, but forcing researchers to choose specific research directions in order to make sure that the investment made in the machines is justified. This trend is most evident and best documented in the case of genomic sequencing, a technology whose development required a high level of investment by governmental agencies \u2013 an investment on which funders expect to see returns, thus pushing researchers to capitalize on the resulting genomic data;<\/li><\/ul>\n<ul><li> the fast-moving technological developments in the relevant field, which makes even very well-established and visible research groups fearful of being left behind or unaware of the latest instruments and techniques on offer.<sup id=\"rdp-ebb-cite_ref-LevinHowDoSci16_33-0\" class=\"reference\"><a href=\"#cite_note-LevinHowDoSci16-33\" rel=\"external_link\">[31]<\/a><\/sup><\/li><\/ul>\n<p>Such widespread insecurity and fears in relation to research environments in the life sciences is not surprising, given the variety of equipment on offer, the high level of technical skill required to use it, the high costs involved in assembling and maintaining an internationally recognized research lab, and the constantly evolving market. Even within well-resourced labs based in prominent and rich institutions, researchers rarely have access to all the technology that they view as potentially relevant to their various projects; and worries around being \"locked-in\" to a given technology, and\/or unable to use it in the most fruitful way, are widespread across highly provisioned research environments. Such worries have arguably grown in parallel to the increased emphasis on transparency and accountability recommended by open science guidelines, and the related explosion of replication experiments pointing to the irreproducibility of many supposedly well-established results. These developments have an enormous potential to improve scientific methods and communication strategies, by eliciting a healthy and necessary preoccupation with producing high-quality, well-justified, intelligible and re-usable results. At the same time, it is important to recognize that open science guidelines and applicability requirements also undermine the implicit trust among peers that so far characterized many areas of biological inquiry, with several researchers confiding to me that they fear being found wanting by colleagues and worry constantly about whether their laboratory set-up and related skills will be recognized as sufficient and well-suited to their line of inquiry.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implications_for_low-resourced_research_environments\">Implications for low-resourced research environments<\/span><\/h2>\n<p>Within high-resourced research environments, there are many mechanisms in place to mitigate the potentially harmful implications of this breakdown in trust, and to turn open science requirements into an opportunity to develop common standards of best practice. First, researchers working in well-funded labs have the means and opportunity to constantly exchange personnel, visits and equipment (and related reagents and materials) with each other, so as to learn from each other and work collaboratively to maintain quality standards in their field. Secondly, researchers based in internationally visible and powerful institutions are in a good position to propose specific (uses of) technology as gold standard for their peers and have the resources to adapt quickly to emerging repertoires, instruments and trends. Furthermore, such researchers typically have access to at least some well-recognized equipment, which they can make accessible to staff from other labs in exchange to access to other tools.\n<\/p><p>These strategies do not always work in the context of an increasingly diverse and globalized research workforce, and particularly not in research locations which are not easily reachable because of their geographical location, and\/or where there are stark inequalities in access to technologies, related infrastructures and materials, and internationally visible and acknowledged collaborative networks. Many biologists are based in contexts where access to the latest and most expensive technology is not guaranteed, financially viable or even relevant \u2013 for instance, because research focuses on areas such as morphology, physiology, developmental biology, botany, immunology and ethology, where access to the most recent genome sequencer may not matter since the production of molecular data may not be the focus of inquiry. Whether or not it affects research practice and outcomes, lack of access to the latest equipment can make researchers insecure on several fronts, including: what they do not have access to, and how important it may be for their work and\/or adherence to international expectations; technical skills that they may lack; and the very reliability and quality of their data, regardless of whether that depends on having the latest equipment. These are similar fears and insecurities to those experienced by researchers working in high-resourced environments. And yet, researchers in low-resourced environments often do not have access to the kinds of buffer available to their better-equipped colleagues, with severe consequences for their publication strategies. In interviews conducted with researchers in South Africa and Kenya in 2014, for instance, it was clear that insecurity around data production methods and access to technology has a strong impact on researchers\u2019 self-confidence and wish to have visibility, share data and publish work internationally.<sup id=\"rdp-ebb-cite_ref-BezuidenhoutBeyond16_12-1\" class=\"reference\"><a href=\"#cite_note-BezuidenhoutBeyond16-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BezuidenhoutBeyond17_34-0\" class=\"reference\"><a href=\"#cite_note-BezuidenhoutBeyond17-34\" rel=\"external_link\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BezuidenhoutTech17_35-0\" class=\"reference\"><a href=\"#cite_note-BezuidenhoutTech17-35\" rel=\"external_link\">[33]<\/a><\/sup>\n<\/p><p>Such findings are not unique nor should they be particularly surprising: scholars in science and technology studies and anthropology have long stressed the role of technology as a marker for identity politics, particularly in the African continent.<sup id=\"rdp-ebb-cite_ref-FergusonGlobal06_36-0\" class=\"reference\"><a href=\"#cite_note-FergusonGlobal06-36\" rel=\"external_link\">[34]<\/a><\/sup> As starkly illustrated recently by work such as Damien Droney's in Ghana<sup id=\"rdp-ebb-cite_ref-DroneyIronies14_37-0\" class=\"reference\"><a href=\"#cite_note-DroneyIronies14-37\" rel=\"external_link\">[35]<\/a><\/sup>, Julie Livingston's in Botswana<sup id=\"rdp-ebb-cite_ref-LivingstonImprovising12_38-0\" class=\"reference\"><a href=\"#cite_note-LivingstonImprovising12-38\" rel=\"external_link\">[36]<\/a><\/sup>, Joanna Crane's in Uganda<sup id=\"rdp-ebb-cite_ref-CraneScrambling13_39-0\" class=\"reference\"><a href=\"#cite_note-CraneScrambling13-39\" rel=\"external_link\">[37]<\/a><\/sup> and Abena Dove Osseo-Asare's across West and East Africa<sup id=\"rdp-ebb-cite_ref-Osseo-AsareBitter14_40-0\" class=\"reference\"><a href=\"#cite_note-Osseo-AsareBitter14-40\" rel=\"external_link\">[38]<\/a><\/sup>, popular culture associates being a scientist with owning spectacular equipment, and this perception filters down to researchers themselves. Equipment is the most visible and concrete marker of wealth in a lab, and it is often interpreted as a signal of the extent to which a research environment in a low-income country can aspire to produce research comparable in quality and significance to that produced by a high-resourced lab. Technology thus becomes a marker for inclusion and a symbol of being part of the Western world in some way \u2013 taking distance from the identity of \"African scientist\" which many researchers find cumbersome and problematic in their dealings with international publishing outlets, funders and institutions. This contributes to the already unequal championing of home-grown scientific approaches and techniques vis-\u00e0-vis methods, concepts and questions imported from the Global North, despite the existence of research areas that are less dependent on expensive machinery and more on elements commonly found across low-resourced environments, such as manpower, expertise and access to specific locations or natural resources.<sup id=\"rdp-ebb-cite_ref-KellyTheWild17_41-0\" class=\"reference\"><a href=\"#cite_note-KellyTheWild17-41\" rel=\"external_link\">[39]<\/a><\/sup>\n<\/p><p>These considerations, which of course apply more widely than African science and potentially include all research conducted in low-resourced environments, bring me to the conclusion that using references to specific technology as proxies for data quality has at least four problematic implications:\n<\/p>\n<dl><dd>1. <i>It may act as an incentive to reduce diversity and creativity in research approaches, by encouraging standardization and the use of the same techniques and technologies regardless of the research context.<\/i><\/dd><\/dl>\n<p>This situation is troublesome when considering the sophisticated relationship between the goals and interests of researchers at different locations, the specific characteristics of each target system in biology, and the methods devised to study those systems; factors which vary widely and can be combined in myriads of ways, giving rise to countless different ways to conduct and validate research. It is also problematic when considering research environments that cannot avail themselves of the latest software or instrument but which are nevertheless producing high-quality data of potential biological significance \u2013 sometimes because of the materials they have access to, sometimes because of their innovative conceptual or methodological approach, and sometimes because they are targeting questions and phenomena of little interest to researchers based elsewhere.\n<\/p>\n<dl><dd>2. <i>It leads to widespread mistrust and fear of openness, particularly when it comes to the sharing of research data.<\/i><\/dd><\/dl>\n<p>All too often, researchers working in low-resourced environments are afraid that lack of access to the latest technologies will affect the quality and reliability of their data and will make their findings vulnerable to criticisms by better-resourced peers. Disparity in access to technologies also affects the speed and efficiency with which data being shared are analyzed, giving researchers based in well-equipped labs the opportunity to analyze and publish on data produced in low-resourced conditions much faster than the original data producers (under the current evaluation regimes, which privilege publication of papers over data production, this is equivalent to being scooped). These fears can result in researchers being unwilling to share their data and\/or to describe the specific circumstances and tools through which they were obtained.\n<\/p>\n<dl><dd>3. <i>It reinforces systematic disadvantage among labs that do not have access to expensive resources.<\/i><\/dd><\/dl>\n<p>This may be the result of researchers' own reluctance to acquire international partners who could question their methods, and\/or to disclose their set-ups (as in the previous point). It may also arise due to the insidious power that assumptions around what counts as a good research environment have within academic structures, evaluation panels and editorial boards of international journals. It is no secret that researchers located in highly reputable institutions have less trouble having their papers accepted for peer review at top-level journals such as <i>Nature<\/i> and <i>Science<\/i>. Similarly, many national policies explicitly ask researchers to emulate the working practices of what are typically regarded as scientific leaders at top Western institutions.\n<\/p>\n<dl><dd>4. <i>It encourages misunderstandings and miscommunication between research data producers and users.<\/i><\/dd><\/dl>\n<p>People who do not articulate the differences between their environments, or feel compelled to minimize them in the name of implicit good standards for \"best practice,\" are at risk of miscommunications and misunderstandings, leading to breakdown in collaborations, problems in interpreting results and difficulties in replicating experiments.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion:_Fostering_critical_engagement_with_data_quality\">Conclusion: Fostering critical engagement with data quality<\/span><\/h2>\n<p>In this paper, I pointed to data quality assessment as crucial to international research collaboration and advancements in the age of open data. At the same time, I warned that the push towards open data, which involves an increasing emphasis on standard data formats and tools for data sharing, is affected by the extensive commercialization of lab equipment and technologies for data dissemination. These elements risk to create a situation where data quality is assessed on the basis of the technologies being employed, rather than the fit between data, methods, materials available and research questions being asked. By contrast, research strategies are typically fine-tuned to the specific questions that researchers wish to pursue and to the phenomena that they wish to study, and such fine-tuning is conducive to research outputs that are credible, well-justified and innovative in their approach and significance. There is thus a wide variety of models for what may count as \"best practice,\" \"adequate data stewardship\" and \"good research environments,\" whose relevance depends on the specific situations of inquiry in which researchers operate. A molecular biology lab with the latest equipment based at Harvard or Cambridge needs not be the standard against which all research set-ups around the world are set, and it should certainly not be implicitly taken to play that role. What type of experimental set-up fits which research project is a contextual matter, depending on many factors including the research questions and approach that is taken, the expertise of the researchers in question, the social dynamics within the group and its international collaborations, and the institutional support, infrastructures and materials available to researchers.\n<\/p><p>This does not mean that researchers working under very different conditions should not talk with each other and exchange tips for improving their environment and working habits. Quite the contrary: acknowledging diversity is an important step towards making such conversation more meaningful and fruitful, as long as it involves challenging the presumption (often unjustified, as the research above demonstrates) that researchers working within the same field actually mean the same when using similar terminologies, and should be constrained in the same ways regardless of the specificities of their working environment.\n<\/p><p>It is imperative that researchers, policy-makers and funders engaged in debates around data quality take these dimensions into account, particularly when thinking about implementing open data practices in low-resourced research environments. The sharing of data typically relies on the ability to use sophisticated data formats and digital data infrastructures, and thus to keep up with the fast pace of technological change associated to such data sharing tools and standards. This becomes problematic given the importance of storing and disseminating multiple data formats, non-digital sources (as in \"old-fashioned\" paper archives) and data produced by different versions of the same software, which helps to embrace the variety of work carried out in the sciences globally (including both low-resourced laboratories and the so-called \"long tail of science\"). Also, it is crucial to enable researchers to develop their projects whether or not they avail themselves of the latest technology, and hence to consider and assess when such technology is needed, and for which purposes.\n<\/p><p>Thus, open data initiatives should be aware of the implications of endorsing specific types of technologies (whether hardware, software or specific laboratory instruments) as markers of research quality. Debates around open data should include explicit and field-specific reflections around the relation between data, research instruments and methods, where researchers clearly articulate their assumptions on what constitutes \"best practice,\" who sets a model for such work, and whether such assumptions are realistic and warranted in light of their own research experiences. This type of articulation is a precious tool for research advancement, since it would encourage confrontation and dialogue at the international level around what quality standards are desirable for data, and with respect to which uses and research goals. These reflexive exercises could be of great value in an ever-globalized and diverse scientific landscape, where the specificity of locations, methods and interests characterizing each research community needs to be documented as essential metadata. In the absence of such critical engagement, open data guidelines risk to dismiss or obscure researchers' situated knowledge and practices (as well as the diversity of fundamental research carried out around the world<sup id=\"rdp-ebb-cite_ref-RochmyaningsihTheDevelop16_42-0\" class=\"reference\"><a href=\"#cite_note-RochmyaningsihTheDevelop16-42\" rel=\"external_link\">[40]<\/a><\/sup>), and instead appeal to politically charged and potentially damaging assumptions about what constitutes \"best practice.\"\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Footnotes\">Footnotes<\/span><\/h2>\n<div class=\"reflist\" style=\"list-style-type: lower-alpha;\">\n<ol class=\"references\">\n<li id=\"cite_note-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-13\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\">The empirical research for this paper was carried out by me within research sites in Wales, Britain, the United States, Germany and Belgium, and by Louise Bezuidenhout within sites in South Africa and Kenya (for more details on the latter research and related methods, see the paper by Bezuidenhout in this special issue). Given the sensitive nature of the interview materials, the raw data underpinning this paper cannot be openly disseminated; however, a digested and anonymized version of the data is provided on Figshare.<sup id=\"rdp-ebb-cite_ref-BezuidenhoutBeyond16_12-0\" class=\"reference\"><a href=\"#cite_note-BezuidenhoutBeyond16-12\" rel=\"external_link\">[12]<\/a><\/sup><\/span>\n<\/li>\n<li id=\"cite_note-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-19\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\">It has also been argued that data quality does not matter within big data collections, because existing data can be triangulated with other datasets documenting the same phenomenon, and datasets that corroborate each other can justifiably be viewed as more reliable.<sup id=\"rdp-ebb-cite_ref-Mayer-Sch.C3.B6nbergerBigData13_16-0\" class=\"reference\"><a href=\"#cite_note-Mayer-Sch.C3.B6nbergerBigData13-16\" rel=\"external_link\">[15]<\/a><\/sup> Against this view, myself and others pointed out that triangulation only works when there are enough datasets that document the same phenomenon from different angles, which is not always the case in scientific research.<sup id=\"rdp-ebb-cite_ref-LeonelliWhatDiff14_17-0\" class=\"reference\"><a href=\"#cite_note-LeonelliWhatDiff14-17\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CaludeTheDeluge16_18-0\" class=\"reference\"><a href=\"#cite_note-CaludeTheDeluge16-18\" rel=\"external_link\">[17]<\/a><\/sup><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Ethics_and_consent\">Ethics and consent<\/span><\/h2>\n<p>The interviews and ethnographies used as empirical source in this paper have received approval by the Social Science Ethics Committee of the University of Exeter, in relation to the projects \"Beyond the Digital Divide\" and \"The Epistemology of Data-Intensive Science.\" All participants signed a consent form. Their contributions are anonymized and their confidentiality is fully respected.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This research was funded by the European Research Council grant award 335925 (\"The Epistemology of Data Science\"), the Leverhulme Trust Grant number RPG-2013-153 (\"Beyond the Digital Divide\"), and the Australian Research Council, Discovery Project DP160102989 (\"Organisms and Us\"). The author gratefully acknowledges input from the Data Studies group at Exeter, and particularly Louise Bezuidenhout, Brian Rappert and Ann Kelly as co-team members of the Leverhulme Project; participants to the \"Pacing Science\" workshop that took place at the University of Exeter in May 2016, where this paper was first presented, and particularly Linsey McGoey and Simon Hodson; members of the Global Young Academy \"Open Science\" and \"Global Access to Research Software\" working groups, especially Abdullah Shams Bin Tariq and Martin Dominik; members of the Knowledge\/Value research network, particularly Kaushik Sunder Rajan and Kris Peterson; and the dozens of researchers who spared time and effort to discuss their methods, outputs and working conditions with me and my colleagues.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The author has no competing interests to declare.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-CaiTheChallenges15-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CaiTheChallenges15_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-CaiTheChallenges15_1-1\" rel=\"external_link\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-CaiTheChallenges15_1-2\" rel=\"external_link\">1.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cai, L.; Zhu, Y. (2015). \"The challenges of data quality and data quality assessment in the big data era\". <i>Data Science Journal<\/i> <b>14<\/b>: 2. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5334%2Fdsj-2015-002\" target=\"_blank\">10.5334\/dsj-2015-002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+challenges+of+data+quality+and+data+quality+assessment+in+the+big+data+era&rft.jtitle=Data+Science+Journal&rft.aulast=Cai%2C+L.%3B+Zhu%2C+Y.&rft.au=Cai%2C+L.%3B+Zhu%2C+Y.&rft.date=2015&rft.volume=14&rft.pages=2&rft_id=info:doi\/10.5334%2Fdsj-2015-002&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OssorioBodies11-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OssorioBodies11_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ossorio, P.N. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3984581\" target=\"_blank\">\"Bodies of data: Genomic data and bioscience data sharing\"<\/a>. <i>Social Research<\/i> <b>78<\/b> (3): 907-932. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3984581\/\" target=\"_blank\">PMC3984581<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24733955\" target=\"_blank\">24733955<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3984581\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3984581<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bodies+of+data%3A+Genomic+data+and+bioscience+data+sharing&rft.jtitle=Social+Research&rft.aulast=Ossorio%2C+P.N.&rft.au=Ossorio%2C+P.N.&rft.date=2011&rft.volume=78&rft.issue=3&rft.pages=907-932&rft_id=info:pmc\/PMC3984581&rft_id=info:pmid\/24733955&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3984581&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BorgmanTheCon12-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BorgmanTheCon12_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Borgman, C.L. (2012). \"The conundrum of sharing research data\". <i>Journal for the Association for Information Science and Technology<\/i> <b>63<\/b> (6): 1059\u20131078. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fasi.22634\" target=\"_blank\">10.1002\/asi.22634<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+conundrum+of+sharing+research+data&rft.jtitle=Journal+for+the+Association+for+Information+Science+and+Technology&rft.aulast=Borgman%2C+C.L.&rft.au=Borgman%2C+C.L.&rft.date=2012&rft.volume=63&rft.issue=6&rft.pages=1059%E2%80%931078&rft_id=info:doi\/10.1002%2Fasi.22634&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonelliData16-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonelliData16_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Leonelli, S. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/D\/bo24957334.html\" target=\"_blank\"><i>Data-Centric Biology: A Philosophical Study<\/i><\/a>. University of Chicago Press. pp. 288. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780226416472<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/D\/bo24957334.html\" target=\"_blank\">http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/D\/bo24957334.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Data-Centric+Biology%3A+A+Philosophical+Study&rft.aulast=Leonelli%2C+S.&rft.au=Leonelli%2C+S.&rft.date=2016&rft.pages=pp.%26nbsp%3B288&rft.pub=University+of+Chicago+Press&rft.isbn=9780226416472&rft_id=http%3A%2F%2Fpress.uchicago.edu%2Fucp%2Fbooks%2Fbook%2Fchicago%2FD%2Fbo24957334.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AllisonRepro16-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AllisonRepro16_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Allison, D.B.; Brown, A.W.; George, B.J.; Kaiser, K.A.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4831566\" target=\"_blank\">\"Reproducibility: A tragedy of errors\"<\/a>. <i>Nature<\/i> <b>530<\/b> (7588): 27\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2F530027a\" target=\"_blank\">10.1038\/530027a<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4831566\/\" target=\"_blank\">PMC4831566<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26842041\" target=\"_blank\">26842041<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4831566\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4831566<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reproducibility%3A+A+tragedy+of+errors&rft.jtitle=Nature&rft.aulast=Allison%2C+D.B.%3B+Brown%2C+A.W.%3B+George%2C+B.J.%3B+Kaiser%2C+K.A.&rft.au=Allison%2C+D.B.%3B+Brown%2C+A.W.%3B+George%2C+B.J.%3B+Kaiser%2C+K.A.&rft.volume=530&rft.issue=7588&rft.pages=27%E2%80%939&rft_id=info:doi\/10.1038%2F530027a&rft_id=info:pmc\/PMC4831566&rft_id=info:pmid\/26842041&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4831566&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PulvererRepro15-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PulvererRepro15_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pulverer, B.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4682652\" target=\"_blank\">\"Reproducibility blues\"<\/a>. <i>EMBO Journal<\/i> <b>34<\/b> (22): 2721-4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.15252%2Fembj.201570090\" target=\"_blank\">10.15252\/embj.201570090<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4682652\/\" target=\"_blank\">PMC4682652<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26538323\" target=\"_blank\">26538323<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4682652\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4682652<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reproducibility+blues&rft.jtitle=EMBO+Journal&rft.aulast=Pulverer%2C+B.&rft.au=Pulverer%2C+B.&rft.volume=34&rft.issue=22&rft.pages=2721-4&rft_id=info:doi\/10.15252%2Fembj.201570090&rft_id=info:pmc\/PMC4682652&rft_id=info:pmid\/26538323&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4682652&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NowotnyNaked11-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NowotnyNaked11_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Nowotny, H.; Testa, G. (2011). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/mitpress.mit.edu\/books\/naked-genes\" target=\"_blank\"><i>Naked Genes<\/i><\/a>. The MIT Press. pp. 152. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780262014939<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/mitpress.mit.edu\/books\/naked-genes\" target=\"_blank\">https:\/\/mitpress.mit.edu\/books\/naked-genes<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Naked+Genes&rft.aulast=Nowotny%2C+H.%3B+Testa%2C+G.&rft.au=Nowotny%2C+H.%3B+Testa%2C+G.&rft.date=2011&rft.pages=pp.%26nbsp%3B152&rft.pub=The+MIT+Press&rft.isbn=9780262014939&rft_id=https%3A%2F%2Fmitpress.mit.edu%2Fbooks%2Fnaked-genes&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-M.C3.BCller-WilleACult12-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-M.C3.BCller-WilleACult12_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">M\u00fcller-Wille, S.W.; Rheinberger, H.J. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.press.uchicago.edu\/ucp\/books\/book\/chicago\/C\/bo8787518.html\" target=\"_blank\"><i>A Cultural History of Heredity<\/i><\/a>. University of Chicago Press. pp. 288. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780226545721<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.press.uchicago.edu\/ucp\/books\/book\/chicago\/C\/bo8787518.html\" target=\"_blank\">http:\/\/www.press.uchicago.edu\/ucp\/books\/book\/chicago\/C\/bo8787518.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=A+Cultural+History+of+Heredity&rft.aulast=M%C3%BCller-Wille%2C+S.W.%3B+Rheinberger%2C+H.J.&rft.au=M%C3%BCller-Wille%2C+S.W.%3B+Rheinberger%2C+H.J.&rft.date=2012&rft.pages=pp.%26nbsp%3B288&rft.pub=University+of+Chicago+Press&rft.isbn=9780226545721&rft_id=http%3A%2F%2Fwww.press.uchicago.edu%2Fucp%2Fbooks%2Fbook%2Fchicago%2FC%2Fbo8787518.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarnesGenomes09-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BarnesGenomes09_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Barnes, B.; Dupr\u00e9, J. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/G\/bo5705879.html\" target=\"_blank\"><i>Genomes and What to Make of Them<\/i><\/a>. University of Chicago Press. pp. 288. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780226172965<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/G\/bo5705879.html\" target=\"_blank\">http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/G\/bo5705879.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Genomes+and+What+to+Make+of+Them&rft.aulast=Barnes%2C+B.%3B+Dupr%C3%A9%2C+J.&rft.au=Barnes%2C+B.%3B+Dupr%C3%A9%2C+J.&rft.date=2009&rft.pages=pp.%26nbsp%3B288&rft.pub=University+of+Chicago+Press&rft.isbn=9780226172965&rft_id=http%3A%2F%2Fpress.uchicago.edu%2Fucp%2Fbooks%2Fbook%2Fchicago%2FG%2Fbo5705879.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Dupr.C3.A9Processes12-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Dupr.C3.A9Processes12_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Dupr\u00e9, J. (2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/global.oup.com\/academic\/product\/processes-of-life-9780199691982?cc=us&lang=en&\" target=\"_blank\"><i>Processes of Life<\/i><\/a>. Oxford University Press. pp. 320. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780199691982<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/global.oup.com\/academic\/product\/processes-of-life-9780199691982?cc=us&lang=en&\" target=\"_blank\">https:\/\/global.oup.com\/academic\/product\/processes-of-life-9780199691982?cc=us&lang=en&<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Processes+of+Life&rft.aulast=Dupr%C3%A9%2C+J.&rft.au=Dupr%C3%A9%2C+J.&rft.date=2012&rft.pages=pp.%26nbsp%3B320&rft.pub=Oxford+University+Press&rft.isbn=9780199691982&rft_id=https%3A%2F%2Fglobal.oup.com%2Facademic%2Fproduct%2Fprocesses-of-life-9780199691982%3Fcc%3Dus%26lang%3Den%26&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-M.C3.BCller-WilleTheGene17-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-M.C3.BCller-WilleTheGene17_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">M\u00fcller-Wille, S.W.; Rheinberger, H.J. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/G\/bo20952390.html\" target=\"_blank\"><i>The Gene: From Genetics to Postgenomics<\/i><\/a>. University of Chicago Press. pp. 176. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780226510002<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/G\/bo20952390.html\" target=\"_blank\">http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/G\/bo20952390.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Gene%3A+From+Genetics+to+Postgenomics&rft.aulast=M%C3%BCller-Wille%2C+S.W.%3B+Rheinberger%2C+H.J.&rft.au=M%C3%BCller-Wille%2C+S.W.%3B+Rheinberger%2C+H.J.&rft.date=2017&rft.pages=pp.%26nbsp%3B176&rft.pub=University+of+Chicago+Press&rft.isbn=9780226510002&rft_id=http%3A%2F%2Fpress.uchicago.edu%2Fucp%2Fbooks%2Fbook%2Fchicago%2FG%2Fbo20952390.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BezuidenhoutBeyond16-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BezuidenhoutBeyond16_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-BezuidenhoutBeyond16_12-1\" rel=\"external_link\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bezuidenhout, L.; Rappert, B.; Leonelli, S.; Kelly, A.H. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/figshare.com\/articles\/Beyond_the_Digital_Divide_Sharing_Research_Data_across_Developing_and_Developed_Countries\/3203809\/1\" target=\"_blank\">\"Beyond the Digital Divide: Sharing Research Data across Developing and Developed Countries\"<\/a>. figshare<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/figshare.com\/articles\/Beyond_the_Digital_Divide_Sharing_Research_Data_across_Developing_and_Developed_Countries\/3203809\/1\" target=\"_blank\">https:\/\/figshare.com\/articles\/Beyond_the_Digital_Divide_Sharing_Research_Data_across_Developing_and_Developed_Countries\/3203809\/1<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Beyond+the+Digital+Divide%3A+Sharing+Research+Data+across+Developing+and+Developed+Countries&rft.atitle=&rft.aulast=Bezuidenhout%2C+L.%3B+Rappert%2C+B.%3B+Leonelli%2C+S.%3B+Kelly%2C+A.H.&rft.au=Bezuidenhout%2C+L.%3B+Rappert%2C+B.%3B+Leonelli%2C+S.%3B+Kelly%2C+A.H.&rft.date=2016&rft.pub=figshare&rft_id=https%3A%2F%2Ffigshare.com%2Farticles%2FBeyond_the_Digital_Divide_Sharing_Research_Data_across_Developing_and_Developed_Countries%2F3203809%2F1&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FloridiThePhil14-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FloridiThePhil14_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Floridi, L.; Illari, P. (2014). <i>The Philosophy of Information Quality<\/i>. Springer International Publishing. pp. 315. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-07121-3\" target=\"_blank\">10.1007\/978-3-319-07121-3<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9783319071213.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Philosophy+of+Information+Quality&rft.aulast=Floridi%2C+L.%3B+Illari%2C+P.&rft.au=Floridi%2C+L.%3B+Illari%2C+P.&rft.date=2014&rft.pages=pp.%26nbsp%3B315&rft.pub=Springer+International+Publishing&rft_id=info:doi\/10.1007%2F978-3-319-07121-3&rft.isbn=9783319071213&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonelliLocating16-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonelliLocating16_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonelli, S.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5124067\" target=\"_blank\">\"Locating ethics in data science: Responsibility and accountability in global and distributed knowledge production systems\"<\/a>. <i>Philosophical Transactions of the Royal Society A<\/i> <b>374<\/b> (2083): 20160122. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1098%2Frsta.2016.0122\" target=\"_blank\">10.1098\/rsta.2016.0122<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5124067\/\" target=\"_blank\">PMC5124067<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28336799\" target=\"_blank\">28336799<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5124067\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5124067<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Locating+ethics+in+data+science%3A+Responsibility+and+accountability+in+global+and+distributed+knowledge+production+systems&rft.jtitle=Philosophical+Transactions+of+the+Royal+Society+A&rft.aulast=Leonelli%2C+S.&rft.au=Leonelli%2C+S.&rft.volume=374&rft.issue=2083&rft.pages=20160122&rft_id=info:doi\/10.1098%2Frsta.2016.0122&rft_id=info:pmc\/PMC5124067&rft_id=info:pmid\/28336799&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5124067&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mayer-Sch.C3.B6nbergerBigData13-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Mayer-Sch.C3.B6nbergerBigData13_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Mayer-Sch\u00f6nberger, V., Cukier, K. (2013). <i>Big Data: A Revolution That Will Transform How We Live, Work and Think<\/i>. John Murray. pp. 256. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781848547926.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Big+Data%3A+A+Revolution+That+Will+Transform+How+We+Live%2C+Work+and+Think&rft.aulast=Mayer-Sch%C3%B6nberger%2C+V.%2C+Cukier%2C+K.&rft.au=Mayer-Sch%C3%B6nberger%2C+V.%2C+Cukier%2C+K.&rft.date=2013&rft.pages=pp.%26nbsp%3B256&rft.pub=John+Murray&rft.isbn=9781848547926&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonelliWhatDiff14-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonelliWhatDiff14_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonelli, S.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4340542\" target=\"_blank\">\"What Difference Does Quantity Make? On the Epistemology of Big Data in Biology\"<\/a>. <i>Big Data and Society<\/i> <b>1<\/b> (1). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F2053951714534395\" target=\"_blank\">10.1177\/2053951714534395<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4340542\/\" target=\"_blank\">PMC4340542<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25729586\" target=\"_blank\">25729586<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4340542\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4340542<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+Difference+Does+Quantity+Make%3F+On+the+Epistemology+of+Big+Data+in+Biology&rft.jtitle=Big+Data+and+Society&rft.aulast=Leonelli%2C+S.&rft.au=Leonelli%2C+S.&rft.volume=1&rft.issue=1&rft_id=info:doi\/10.1177%2F2053951714534395&rft_id=info:pmc\/PMC4340542&rft_id=info:pmid\/25729586&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4340542&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaludeTheDeluge16-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CaludeTheDeluge16_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Calude, C.S.; Longo, G.. \"The Deluge of Spurious Correlations in Big Data\". <i>Foundations of Science<\/i> <b>21<\/b>: 1\u201318. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10699-016-9489-4\" target=\"_blank\">10.1007\/s10699-016-9489-4<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Deluge+of+Spurious+Correlations+in+Big+Data&rft.jtitle=Foundations+of+Science&rft.aulast=Calude%2C+C.S.%3B+Longo%2C+G.&rft.au=Calude%2C+C.S.%3B+Longo%2C+G.&rft.volume=21&rft.pages=1%E2%80%9318&rft_id=info:doi\/10.1007%2Fs10699-016-9489-4&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoreyThePeer16-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoreyThePeer16_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Morey, R.D.; Chambers, C.D.; Etchells, P.J. et al.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4736937\" target=\"_blank\">\"The Peer Reviewers' Openness Initiative: Incentivizing open research practices through peer review\"<\/a>. <i>Royal Society Open Science<\/i> <b>3<\/b> (1): 150547. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1098%2Frsos.150547\" target=\"_blank\">10.1098\/rsos.150547<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4736937\/\" target=\"_blank\">PMC4736937<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26909182\" target=\"_blank\">26909182<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4736937\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4736937<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Peer+Reviewers%27+Openness+Initiative%3A+Incentivizing+open+research+practices+through+peer+review&rft.jtitle=Royal+Society+Open+Science&rft.aulast=Morey%2C+R.D.%3B+Chambers%2C+C.D.%3B+Etchells%2C+P.J.+et+al.&rft.au=Morey%2C+R.D.%3B+Chambers%2C+C.D.%3B+Etchells%2C+P.J.+et+al.&rft.volume=3&rft.issue=1&rft.pages=150547&rft_id=info:doi\/10.1098%2Frsos.150547&rft_id=info:pmc\/PMC4736937&rft_id=info:pmid\/26909182&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4736937&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonelliHowTheGene11-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonelliHowTheGene11_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonelli, S.; Diehl, A.D.; Christie, K.R. et al.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3166943\" target=\"_blank\">\"How the gene ontology evolves\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b>: 325. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-325\" target=\"_blank\">10.1186\/1471-2105-12-325<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3166943\/\" target=\"_blank\">PMC3166943<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21819553\" target=\"_blank\">21819553<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3166943\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3166943<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+the+gene+ontology+evolves&rft.jtitle=BMC+Bioinformatics&rft.aulast=Leonelli%2C+S.%3B+Diehl%2C+A.D.%3B+Christie%2C+K.R.+et+al.&rft.au=Leonelli%2C+S.%3B+Diehl%2C+A.D.%3B+Christie%2C+K.R.+et+al.&rft.volume=12&rft.pages=325&rft_id=info:doi\/10.1186%2F1471-2105-12-325&rft_id=info:pmc\/PMC3166943&rft_id=info:pmid\/21819553&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3166943&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlakeGeneOnt15-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlakeGeneOnt15_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blake, J.A.; Christie, K.R.; Dolan, M.E. et al.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383973\" target=\"_blank\">\"Gene Ontology Consortium: Going forward\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (DB1): D1049-56. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1179\" target=\"_blank\">10.1093\/nar\/gku1179<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383973\/\" target=\"_blank\">PMC4383973<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25428369\" target=\"_blank\">25428369<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383973\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383973<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+Ontology+Consortium%3A+Going+forward&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Blake%2C+J.A.%3B+Christie%2C+K.R.%3B+Dolan%2C+M.E.+et+al.&rft.au=Blake%2C+J.A.%3B+Christie%2C+K.R.%3B+Dolan%2C+M.E.+et+al.&rft.volume=43&rft.issue=DB1&rft.pages=D1049-56&rft_id=info:doi\/10.1093%2Fnar%2Fgku1179&rft_id=info:pmc\/PMC4383973&rft_id=info:pmid\/25428369&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383973&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McDowallPomBase15-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McDowallPomBase15_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McDowall, M.D.; Harris, M.A.; Lock, A. et al.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383888\" target=\"_blank\">\"PomBase 2015: Updates to the fission yeast database\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (DB1): D656-61. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1040\" target=\"_blank\">10.1093\/nar\/gku1040<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383888\/\" target=\"_blank\">PMC4383888<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25361970\" target=\"_blank\">25361970<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383888\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383888<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PomBase+2015%3A+Updates+to+the+fission+yeast+database&rft.jtitle=Nucleic+Acids+Research&rft.aulast=McDowall%2C+M.D.%3B+Harris%2C+M.A.%3B+Lock%2C+A.+et+al.&rft.au=McDowall%2C+M.D.%3B+Harris%2C+M.A.%3B+Lock%2C+A.+et+al.&rft.volume=43&rft.issue=DB1&rft.pages=D656-61&rft_id=info:doi\/10.1093%2Fnar%2Fgku1040&rft_id=info:pmc\/PMC4383888&rft_id=info:pmid\/25361970&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383888&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KambatlaTrends14-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KambatlaTrends14_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kambatla, K.; Kollias, G.; Kumar, V.; Grama, A.. \"Trends in big data analytics\". <i>Journal of Parallel and Distributed Computing<\/i> <b>74<\/b> (7): 2561-2573. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpdc.2014.01.003\" target=\"_blank\">10.1016\/j.jpdc.2014.01.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Trends+in+big+data+analytics&rft.jtitle=Journal+of+Parallel+and+Distributed+Computing&rft.aulast=Kambatla%2C+K.%3B+Kollias%2C+G.%3B+Kumar%2C+V.%3B+Grama%2C+A.&rft.au=Kambatla%2C+K.%3B+Kollias%2C+G.%3B+Kumar%2C+V.%3B+Grama%2C+A.&rft.volume=74&rft.issue=7&rft.pages=2561-2573&rft_id=info:doi\/10.1016%2Fj.jpdc.2014.01.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AnkenyValuing15-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AnkenyValuing15_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ankeny, R.A.; Leonelli, S. (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.dukeupress.edu\/postgenomics\" target=\"_blank\">\"Valuing Data in Postgenomic Biology: How Data Donation and Curation Practices Challenge the Scientific Publication System\"<\/a>. <i>Postgenomics: Perspectives on Biology after the Genome<\/i>. Duke University Press. pp. 126\u2013149. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780822358947<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.dukeupress.edu\/postgenomics\" target=\"_blank\">https:\/\/www.dukeupress.edu\/postgenomics<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Valuing+Data+in+Postgenomic+Biology%3A+How+Data+Donation+and+Curation+Practices+Challenge+the+Scientific+Publication+System&rft.atitle=Postgenomics%3A+Perspectives+on+Biology+after+the+Genome&rft.aulast=Ankeny%2C+R.A.%3B+Leonelli%2C+S.&rft.au=Ankeny%2C+R.A.%3B+Leonelli%2C+S.&rft.date=2015&rft.pages=pp.%26nbsp%3B126%E2%80%93149&rft.pub=Duke+University+Press&rft.isbn=9780822358947&rft_id=https%3A%2F%2Fwww.dukeupress.edu%2Fpostgenomics&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AnkenyRepertoires16-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AnkenyRepertoires16_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ankeny, R.A.; Leonelli, S.. \"Repertoires: A post-Kuhnian perspective on scientific change and collaborative research\". <i>Studies in History and Philosophy of Science<\/i> <b>60<\/b>: 18\u201328. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.shpsa.2016.08.003\" target=\"_blank\">10.1016\/j.shpsa.2016.08.003<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27938718\" target=\"_blank\">27938718<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Repertoires%3A+A+post-Kuhnian+perspective+on+scientific+change+and+collaborative+research&rft.jtitle=Studies+in+History+and+Philosophy+of+Science&rft.aulast=Ankeny%2C+R.A.%3B+Leonelli%2C+S.&rft.au=Ankeny%2C+R.A.%3B+Leonelli%2C+S.&rft.volume=60&rft.pages=18%E2%80%9328&rft_id=info:doi\/10.1016%2Fj.shpsa.2016.08.003&rft_id=info:pmid\/27938718&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrohsConvenience12-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrohsConvenience12_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krohs, U.. \"Convenience experimentation\". <i>Studies in History and Philosophy of Biological and Biomedical Sciences<\/i> <b>43<\/b> (1): 52-7. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.shpsc.2011.10.005\" target=\"_blank\">10.1016\/j.shpsc.2011.10.005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22326072\" target=\"_blank\">22326072<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Convenience+experimentation&rft.jtitle=Studies+in+History+and+Philosophy+of+Biological+and+Biomedical+Sciences&rft.aulast=Krohs%2C+U.&rft.au=Krohs%2C+U.&rft.volume=43&rft.issue=1&rft.pages=52-7&rft_id=info:doi\/10.1016%2Fj.shpsc.2011.10.005&rft_id=info:pmid\/22326072&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonelliRepertoires15-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonelliRepertoires15_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonelli, S.; Ankeny, R.A.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4580990\" target=\"_blank\">\"Repertoires: How to Transform a Project into a Research Community\"<\/a>. <i>BioScience<\/i> <b>65<\/b> (7): 701\u2013708. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbiosci%2Fbiv061\" target=\"_blank\">10.1093\/biosci\/biv061<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4580990\/\" target=\"_blank\">PMC4580990<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26412866\" target=\"_blank\">26412866<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4580990\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4580990<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Repertoires%3A+How+to+Transform+a+Project+into+a+Research+Community&rft.jtitle=BioScience&rft.aulast=Leonelli%2C+S.%3B+Ankeny%2C+R.A.&rft.au=Leonelli%2C+S.%3B+Ankeny%2C+R.A.&rft.volume=65&rft.issue=7&rft.pages=701%E2%80%93708&rft_id=info:doi\/10.1093%2Fbiosci%2Fbiv061&rft_id=info:pmc\/PMC4580990&rft_id=info:pmid\/26412866&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4580990&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RajanBiocapital06-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RajanBiocapital06_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Rajan, K.S. (2006). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.dukeupress.edu\/biocapital\" target=\"_blank\"><i>Biocapital: The Constitution of Postgenomic Life<\/i><\/a>. Duke University Press. pp. 360. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780822337201<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.dukeupress.edu\/biocapital\" target=\"_blank\">https:\/\/www.dukeupress.edu\/biocapital<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Biocapital%3A+The+Constitution+of+Postgenomic+Life&rft.aulast=Rajan%2C+K.S.&rft.au=Rajan%2C+K.S.&rft.date=2006&rft.pages=pp.%26nbsp%3B360&rft.pub=Duke+University+Press&rft.isbn=9780822337201&rft_id=https%3A%2F%2Fwww.dukeupress.edu%2Fbiocapital&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ThayerTopInst16-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ThayerTopInst16_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thayer, A.M. (25 April 2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/cen.acs.org\/articles\/94\/i17\/Top-instrument-firms-2015.html\" target=\"_blank\">\"Top instrument firms in 2015\"<\/a>. <i>C&EN<\/i> <b>94<\/b> (17): 32\u201335<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/cen.acs.org\/articles\/94\/i17\/Top-instrument-firms-2015.html\" target=\"_blank\">http:\/\/cen.acs.org\/articles\/94\/i17\/Top-instrument-firms-2015.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Top+instrument+firms+in+2015&rft.jtitle=C%26EN&rft.aulast=Thayer%2C+A.M.&rft.au=Thayer%2C+A.M.&rft.date=25+April+2016&rft.volume=94&rft.issue=17&rft.pages=32%E2%80%9335&rft_id=http%3A%2F%2Fcen.acs.org%2Farticles%2F94%2Fi17%2FTop-instrument-firms-2015.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RogersMaking07-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RogersMaking07_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rogers, S.; Cambrosio, A. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2347363\" target=\"_blank\">\"Making a new technology work: The standardization and regulation of microarrays\"<\/a>. <i>Yale Journal of Biology and Medicine<\/i> <b>80<\/b> (4): 165-78. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2347363\/\" target=\"_blank\">PMC2347363<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18449388\" target=\"_blank\">18449388<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2347363\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2347363<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Making+a+new+technology+work%3A+The+standardization+and+regulation+of+microarrays&rft.jtitle=Yale+Journal+of+Biology+and+Medicine&rft.aulast=Rogers%2C+S.%3B+Cambrosio%2C+A.&rft.au=Rogers%2C+S.%3B+Cambrosio%2C+A.&rft.date=2007&rft.volume=80&rft.issue=4&rft.pages=165-78&rft_id=info:pmc\/PMC2347363&rft_id=info:pmid\/18449388&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2347363&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KaloramaTheWorld16-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KaloramaTheWorld16_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Kalorama Information (January 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchandmarkets.com\/reports\/3619329\/the-world-market-for-microarrays\" target=\"_blank\">\"The World Market for Microarrays\"<\/a>. Research and Markets<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.researchandmarkets.com\/reports\/3619329\/the-world-market-for-microarrays\" target=\"_blank\">https:\/\/www.researchandmarkets.com\/reports\/3619329\/the-world-market-for-microarrays<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+World+Market+for+Microarrays&rft.atitle=&rft.aulast=Kalorama+Information&rft.au=Kalorama+Information&rft.date=January+2016&rft.pub=Research+and+Markets&rft_id=https%3A%2F%2Fwww.researchandmarkets.com%2Freports%2F3619329%2Fthe-world-market-for-microarrays&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LevinHowDoSci16-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LevinHowDoSci16_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Levin, N.; Leonelli, S. Weckowska, D. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5066505\" target=\"_blank\">\"How Do Scientists Define Openness? Exploring the Relationship Between Open Science Policies and Research Practice\"<\/a>. <i>Bulletin of Science, Technology, & Society<\/i> <b>36<\/b> (2): 128-141. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F0270467616668760\" target=\"_blank\">10.1177\/0270467616668760<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5066505\/\" target=\"_blank\">PMC5066505<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27807390\" target=\"_blank\">27807390<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5066505\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5066505<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+Do+Scientists+Define+Openness%3F+Exploring+the+Relationship+Between+Open+Science+Policies+and+Research+Practice&rft.jtitle=Bulletin+of+Science%2C+Technology%2C+%26+Society&rft.aulast=Levin%2C+N.%3B+Leonelli%2C+S.+Weckowska%2C+D.+et+al.&rft.au=Levin%2C+N.%3B+Leonelli%2C+S.+Weckowska%2C+D.+et+al.&rft.date=2016&rft.volume=36&rft.issue=2&rft.pages=128-141&rft_id=info:doi\/10.1177%2F0270467616668760&rft_id=info:pmc\/PMC5066505&rft_id=info:pmid\/27807390&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5066505&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BezuidenhoutBeyond17-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BezuidenhoutBeyond17_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bezuidenhout, L.; Leonelli, S.; Kelly, A.H., Rappert, B. (2017). \"Beyond the digital divide: Towards a situated approach to open data\". <i>Science and Public Policy<\/i>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fscipol%2Fscw036\" target=\"_blank\">10.1093\/scipol\/scw036<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Beyond+the+digital+divide%3A+Towards+a+situated+approach+to+open+data&rft.jtitle=Science+and+Public+Policy&rft.aulast=Bezuidenhout%2C+L.%3B+Leonelli%2C+S.%3B+Kelly%2C+A.H.%2C+Rappert%2C+B.&rft.au=Bezuidenhout%2C+L.%3B+Leonelli%2C+S.%3B+Kelly%2C+A.H.%2C+Rappert%2C+B.&rft.date=2017&rft_id=info:doi\/10.1093%2Fscipol%2Fscw036&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BezuidenhoutTech17-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BezuidenhoutTech17_35-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bezuidenhout, L. (2017). \"Technology Transfer and True Transformation: Implications for Open Data\". <i>Data Science Journal<\/i> <b>16<\/b>: 26. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5334%2Fdsj-2017-026\" target=\"_blank\">10.5334\/dsj-2017-026<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Technology+Transfer+and+True+Transformation%3A+Implications+for+Open+Data&rft.jtitle=Data+Science+Journal&rft.aulast=Bezuidenhout%2C+L.&rft.au=Bezuidenhout%2C+L.&rft.date=2017&rft.volume=16&rft.pages=26&rft_id=info:doi\/10.5334%2Fdsj-2017-026&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FergusonGlobal06-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FergusonGlobal06_36-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ferguson, J. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/read.dukeupress.edu\/content\/global-shadows\" target=\"_blank\"><i>Global Shadows: Africa in the Neoliberal World Order<\/i><\/a>. Duke University Press. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1215%2F9780822387640\" target=\"_blank\">10.1215\/9780822387640<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780822387640<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/read.dukeupress.edu\/content\/global-shadows\" target=\"_blank\">http:\/\/read.dukeupress.edu\/content\/global-shadows<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Global+Shadows%3A+Africa+in+the+Neoliberal+World+Order&rft.aulast=Ferguson%2C+J.&rft.au=Ferguson%2C+J.&rft.date=2006&rft.pub=Duke+University+Press&rft_id=info:doi\/10.1215%2F9780822387640&rft.isbn=9780822387640&rft_id=http%3A%2F%2Fread.dukeupress.edu%2Fcontent%2Fglobal-shadows&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DroneyIronies14-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DroneyIronies14_37-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Droney, D. (2014). \"Ironies of Laboratory Work during Ghana's Second Age of Optimism\". <i>Cultural Anthropology<\/i> <b>29<\/b> (2): 363\u2013384. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.14506%2Fca29.2.10\" target=\"_blank\">10.14506\/ca29.2.10<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ironies+of+Laboratory+Work+during+Ghana%27s+Second+Age+of+Optimism&rft.jtitle=Cultural+Anthropology&rft.aulast=Droney%2C+D.&rft.au=Droney%2C+D.&rft.date=2014&rft.volume=29&rft.issue=2&rft.pages=363%E2%80%93384&rft_id=info:doi\/10.14506%2Fca29.2.10&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LivingstonImprovising12-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LivingstonImprovising12_38-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Livingston, J. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/read.dukeupress.edu\/content\/improvising-medicine\" target=\"_blank\"><i>Improvising Medicine: An African Oncology Ward in an Emerging Cancer Epidemic<\/i><\/a>. Duke University Press. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1215%2F9780822395768\" target=\"_blank\">10.1215\/9780822395768<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780822395768<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/read.dukeupress.edu\/content\/improvising-medicine\" target=\"_blank\">http:\/\/read.dukeupress.edu\/content\/improvising-medicine<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Improvising+Medicine%3A+An+African+Oncology+Ward+in+an+Emerging+Cancer+Epidemic&rft.aulast=Livingston%2C+J.&rft.au=Livingston%2C+J.&rft.date=2012&rft.pub=Duke+University+Press&rft_id=info:doi\/10.1215%2F9780822395768&rft.isbn=9780822395768&rft_id=http%3A%2F%2Fread.dukeupress.edu%2Fcontent%2Fimprovising-medicine&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CraneScrambling13-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CraneScrambling13_39-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Crane, J.T. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.cornellpress.cornell.edu\/book\/?GCOI=80140100922670\" target=\"_blank\"><i>Scrambling for Africa: AIDS, Expertise, and the Rise of American Global Health Science<\/i><\/a>. Cornell University Press. pp. 224. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780801451959<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cornellpress.cornell.edu\/book\/?GCOI=80140100922670\" target=\"_blank\">http:\/\/www.cornellpress.cornell.edu\/book\/?GCOI=80140100922670<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Scrambling+for+Africa%3A+AIDS%2C+Expertise%2C+and+the+Rise+of+American+Global+Health+Science&rft.aulast=Crane%2C+J.T.&rft.au=Crane%2C+J.T.&rft.date=2013&rft.pages=pp.%26nbsp%3B224&rft.pub=Cornell+University+Press&rft.isbn=9780801451959&rft_id=http%3A%2F%2Fwww.cornellpress.cornell.edu%2Fbook%2F%3FGCOI%3D80140100922670&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Osseo-AsareBitter14-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Osseo-AsareBitter14_40-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Osseo-Asare, A.D. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/B\/bo17031345.html\" target=\"_blank\"><i>Bitter Roots: The Search for Healing Plants in Africa<\/i><\/a>. University of Chicago Press. pp. 288. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780226086163<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/B\/bo17031345.html\" target=\"_blank\">http:\/\/press.uchicago.edu\/ucp\/books\/book\/chicago\/B\/bo17031345.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Bitter+Roots%3A+The+Search+for+Healing+Plants+in+Africa&rft.aulast=Osseo-Asare%2C+A.D.&rft.au=Osseo-Asare%2C+A.D.&rft.date=2014&rft.pages=pp.%26nbsp%3B288&rft.pub=University+of+Chicago+Press&rft.isbn=9780226086163&rft_id=http%3A%2F%2Fpress.uchicago.edu%2Fucp%2Fbooks%2Fbook%2Fchicago%2FB%2Fbo17031345.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KellyTheWild17-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KellyTheWild17_41-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kelly, A.; Lezaun, J. (2017). \"The wild indoors: Room-spaces of scientific inquiry\". <i>Cultural Anthropology<\/i>. \"In press\"<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+wild+indoors%3A+Room-spaces+of+scientific+inquiry&rft.jtitle=Cultural+Anthropology&rft.aulast=Kelly%2C+A.%3B+Lezaun%2C+J.&rft.au=Kelly%2C+A.%3B+Lezaun%2C+J.&rft.date=2017&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RochmyaningsihTheDevelop16-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RochmyaningsihTheDevelop16_42-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rochmyaningsih, D.. \"The developing world needs basic research too\". <i>Nature<\/i> <b>534<\/b> (7605): 7. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2F534007a\" target=\"_blank\">10.1038\/534007a<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27251238\" target=\"_blank\">27251238<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+developing+world+needs+basic+research+too&rft.jtitle=Nature&rft.aulast=Rochmyaningsih%2C+D.&rft.au=Rochmyaningsih%2C+D.&rft.volume=534&rft.issue=7605&rft.pages=7&rft_id=info:doi\/10.1038%2F534007a&rft_id=info:pmid\/27251238&rfr_id=info:sid\/en.wikipedia.org:Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, including regionalizing spelling. In some cases important information was missing from the references, and that information was added. The original article had citations listed alphabetically; they are listed in the order they appear here due to the way the wiki works. In several cases, the original article cited sources inline (e.g., Primiero 2014, Stegenga 2014, and Hilgartner 2017) but failed to list them in the References section, or the article didn't even exist yet (Rappert this issue); these have been omitted here. One URL was broken in the original but updated with the current one in this version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181214181449\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.870 seconds\nReal time usage: 0.892 seconds\nPreprocessor visited node count: 31327\/1000000\nPreprocessor generated node count: 38718\/1000000\nPost\u2010expand include size: 243431\/2097152 bytes\nTemplate argument size: 75301\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 870.966 1 - -total\n 88.11% 767.367 2 - Template:Reflist\n 77.39% 674.003 40 - Template:Citation\/core\n 51.49% 448.486 24 - Template:Cite_journal\n 26.57% 231.426 14 - Template:Cite_book\n 8.70% 75.757 64 - Template:Citation\/identifier\n 7.00% 61.003 1 - Template:Infobox_journal_article\n 6.67% 58.078 1 - Template:Infobox\n 4.09% 35.603 41 - Template:Citation\/make_link\n 3.90% 33.957 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:10160-0!*!*!!en!*!* and timestamp 20181214181448 and revision id 30841\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology\">https:\/\/www.limswiki.org\/index.php\/Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","141c57a5e72031b2191141bec29c26b6_images":[],"141c57a5e72031b2191141bec29c26b6_timestamp":1544811288,"5558150b977a44d9e5f293e9ae7e49a1":{"type":"chapter","title":"1. Big data, informatics, and research","key":"5558150b977a44d9e5f293e9ae7e49a1"}},"link":"https:\/\/www.limswiki.org\/index.php\/Book:LIMSjournal_-_Summer_2017","price_currency":"","price_amount":"","book_size":"","download_url":"https:\/\/www.limsforum.com?ebb_action=book_download&book_id=78043","language":"","cta_button_content":"","toc":[{"type":"chapter","name":"1. Big data, informatics, and research","id":"5558150b977a44d9e5f293e9ae7e49a1","children":[{"type":"article","name":"Global data quality assessment and the situated nature of \u201cbest\u201d research practices in biology (Leonelli 2017)","id":"141c57a5e72031b2191141bec29c26b6","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Global_data_quality_assessment_and_the_situated_nature_of_%E2%80%9Cbest%E2%80%9D_research_practices_in_biology"},{"type":"article","name":"The FAIR Guiding Principles for scientific data management and stewardship (Wilkinson et al. 2016)","id":"e5903ddcc7734415af1d91fcd258da90","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship"}]},{"type":"chapter","name":"2. Bioinformatics","id":"77fbf09bb35e82206c113e9ae59b1b18","children":[{"type":"article","name":"Analyzing the field of bioinformatics with the multi-faceted topic modeling technique (Heo et al. 2017)","id":"55740ce4aa95ae034b56b161256958e3","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_the_field_of_bioinformatics_with_the_multi-faceted_topic_modeling_technique"},{"type":"article","name":"Intervene: A tool for intersection and visualization of multiple gene or genomic region sets (Khan and Mathelier 2017)","id":"dd7402ef5c2b99b24877d5ee4e9d0720","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Intervene:_A_tool_for_intersection_and_visualization_of_multiple_gene_or_genomic_region_sets"}]},{"type":"chapter","name":"3. Health, public health, and clinical informatics","id":"e20bd57b1986b44204a29c8419326fe3","children":[{"type":"article","name":"The state of open-source electronic health record projects: A software anthropology study (Alsaffar et al. 2017)","id":"61508b4d2723dec78d6fb5e85ab3f5c1","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:The_state_of_open-source_electronic_health_record_projects:_A_software_anthropology_study"},{"type":"article","name":"Users\u2019 perspectives on a picture archiving and communication system (PACS): An in-depth study in a teaching hospital in Kuwait (Buabbas et al. 2016)","id":"016989f9a69b54fcd5805d9f8681d6fc","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Users%E2%80%99_perspectives_on_a_picture_archiving_and_communication_system_(PACS):_An_in-depth_study_in_a_teaching_hospital_in_Kuwait"},{"type":"article","name":"The effect of a test ordering software intervention on the prescription of unnecessary laboratory tests - A randomized controlled trial (Martins et al. 2017)","id":"45e8147ab685e78229d6e364a84469a5","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:The_effect_of_a_test_ordering_software_intervention_on_the_prescription_of_unnecessary_laboratory_tests_-_A_randomized_controlled_trial"},{"type":"article","name":"Effective information extraction framework for heterogeneous clinical reports using online machine learning and controlled vocabularies (Zheng et al. 2017)","id":"a67695148af73db7d41c4171d7c86cbc","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Effective_information_extraction_framework_for_heterogeneous_clinical_reports_using_online_machine_learning_and_controlled_vocabularies"}]},{"type":"chapter","name":"4. Laboratory information management systems","id":"65e50ee7a77cb06c5e5b3d5e11b46b54","children":[{"type":"article","name":"Baobab Laboratory Information Management System: Development of an open-source laboratory information management system for biobanking (Bendou et al. 2017)","id":"9e160b0ad749c44fbc6303a7527b6e1e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Baobab_Laboratory_Information_Management_System:_Development_of_an_open-source_laboratory_information_management_system_for_biobanking"},{"type":"article","name":"MASTR-MS: A web-based collaborative laboratory information management system (LIMS) for metabolomics (Hunter et al. 2017)","id":"c1c21b43f0762acc3ad8147824a0e651","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:MASTR-MS:_A_web-based_collaborative_laboratory_information_management_system_(LIMS)_for_metabolomics"},{"type":"article","name":"Selecting a laboratory information management system for biorepositories in low- and middle-income countries: The H3Africa experience and lessons learned (Kyobe et al. 2017)","id":"021382bc6a00c9f9a06aef062ae61eaa","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Selecting_a_laboratory_information_management_system_for_biorepositories_in_low-_and_middle-income_countries:_The_H3Africa_experience_and_lessons_learned"}]},{"type":"chapter","name":"5. Miscellaneous","id":"4dfb7e9d4e03c5d7465343a788befa5d","children":[{"type":"article","name":"A multi-service data management platform for scientific oceanographic products (D\u2019Anca et al. 2017)","id":"e86f9fcd9da316970f965a883c42e462","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:A_multi-service_data_management_platform_for_scientific_oceanographic_products"},{"type":"article","name":"Neuroimaging, genetics, and clinical data sharing in Python using the CubicWeb framework (Grigis et al. 2017)","id":"bce85c098ea6958c92b6dcce94e42565","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Neuroimaging,_genetics,_and_clinical_data_sharing_in_Python_using_the_CubicWeb_framework"}]}],"settings":{"show_cover":"1","show_title":"1","show_subtitle":"0","show_full_title":"1","show_editor":"1","show_editor_pic":"1","show_publisher":"1","show_language":"1","show_size":"1","show_toc":"1","show_content_beneath_cover":"1","cta_button":"1","content_location":"1","toc_links":"disabled","log_in_msg":"<span><\/span> Please log in to read online.","cover_size":"medium"},"title_image":"https:\/\/s3.limsforum.com\/www.limsforum.com\/wp-content\/uploads\/Fig3_DAnca_NatHazEarth2017_17-2.png"}}
Copyright LabLynx Inc. All rights reserved.