Chapter 6 Data Management

6.1 Clinical Data Management

6.1.1 What

Clinical Data Management (CDM) is the collection, cleaning, and management of subject data in compliance with regulatory standards (Krishnankutty_2012)). It is a critical component of clinical research.

6.1.2 Why

The quality of the data generated by the clinical trial will determine the soundness of scientific conclusions formed on the basis of analysis.

6.1.3 How

Data collection and management procedures should be based on sound statistical principles and compliant with the study protocol and regulations. CDM protocol should address key project prerequisites regarding the data collection, storage, protection, retention, and reporting. Potential problems and solutions proposed solutions should be reviewed in detail.

The International Conference On Harmonisation (ICH) Guideline E6(R2) recommends that sponsors of clinical trials “implement systems to manage quality throughout the design, conduct, recording, evaluation, reporting, and archiving of clinical trials”… Sponsors should focus on the “human subject protection and reliability of trial results”… To warrant the quality of management “the methods selected to assure and control the quality of the trial need to be proportionate to the risks inherent in the trial and the importance of the information collected” (E6_R2: 5).

The NIH Intramural Research Program Guidelines section on Clinical Informatics, Data Management, and Protocol Tracking rationale and standard states:

“Collecting clinical data is a complex task that must be integrated into the medical practices of the institution. To monitor the study’s progress and patient safety, data collection is best done as data are generated. Data management organized and supported at the institute level is more efficient and reliable than that left to the individual investigator. There are often unforeseen uses for the kinds of information gathered in the conduct of a clinical trial, and a central database, with appropriate archiving, assures that this information remains the legacy of the institute”

Each institute sponsoring clinical research should develop a central clinical investigations database that maintains all data specified to be collected in the clinical study (either intervention or natural history). The clinical research information system being continually developed by the Clinical Center interfaces with and supports each institute’s clinical research needs. A confederated database will enable information exchange, enabling access to and sharing of clinical and research information among all institutes. The institutes require data-management infrastructures to maintain their central data registries, to enhance existing databases, to provide eligibility checklists, to record patient randomization and entry into their protocols, to provide report generation, data warehousing, and data entry forms, and to monitor data collection.” (NIH_standars_CR)

The NIA Clinical Research Study Investigator’s Toolbox provides guidance with specific reference to clinical trials in aging, and in particular useful tips for data management. Investigators pursuing or conducting federally funded projects should make use of this resource (NIA-Guide MOP).

6.2 Clinical Data Management Definition and Procedures

6.2.1 What

The clinical data management procedures define the methods and dependent activities in which the clinical data is collected and managed. The procedures content should include the methods used to assign and structure participant’s identifiers (ID), the location of the ID logs, the types of data collection instruments used, a description of how the data is captured/completed, reviewed, cleaned, and subsequently stored/archived.

6.2.2 Why

Data management procedures that are compliant with the trial’s protocol, good clinical practice (GCP), regulatory requirements, and undergo regular process audits assures the quality of data necessary to execute the planned analysis.

6.2.3 How

The data management procedures are often incorporated as a subsection within the Manual of Operations (MOO) or Standard Operating Procedures (SOP). These are executed by project personnel whose portfolio of work includes data management, and who have appropriate professional training and credentials in data science broadly defined. IFAR encourages all investigators and managers to author a MOO or SOP in order to document the project operations including the data management activities (IFAR_sensitive_data_security). As noted above, the NIA toolbox provides general guidance and specific requirements for clinical trials conducted in the field of aging. Harvard Catalyst has a detailed planning checklist that can assist researchers describe the data for protection, IRB submission, and regulatory compliance (Harvard_Catalyst: 5).

Extensive and diverse guidance and examples in other fields are also available online. Some well-executed examples are featured the National Institute of Dental and Craniofacial Research (NIDCR) and Boston Children’s Hospital (BCH) Clinical Research Center websites.

6.2.3.1 Systems for data collection and management

Clinical trials may use paper-based or electronic data capture, with most studies today using the latter. An electronic data capture (EDC) system will be designed to facilitate data collection, often using web-based technology. A Clinical Data Management Systems (CDMS) comprises the software tools for data maintenance, quality assurance and reporting. EDC and CDMS may be combined under a single informatics environment.

ICH guideline E6(R2) recommends that the informatics environment allows for both data changes (edits) and detailed trail of audits, data, and editorial information. It also recommends maintaining an SOP for using these systems and a security system that prevents unauthorized access to the data (E6_R2: 5.5).

When describing the data collection system in applications or protocols, data management personnel should define:

  • The system specifications: the version number, security system, audit trail, data entry access, the annotated case report forms (CFR), data validation, system failure mitigation plan, and electronic signature features.

  • User acceptance testing: the proper function of the system. This includes the customized functions, validation checks, the proper assignment of site and/or participant identifications numbers, the cross confirmation of time to events variable delivery, confirmation of data completion, the proper functionality of electronic (e.g. e-mail) alerts, as well as the proper function of role assignments, data exports/imports, and system reports.

    Detail concerning specific aspects of procedures governing, and design of, EDC and CDMS are provided in the sections below.

6.2.3.1.1 EDC/CDM systems available at IFAR

See Appendix I

6.2.3.2 Data entry procedures

These outline rules for valid data capture. They include the method for determining that a specific form has been completed, for data cleaning and quality assurance, storage procedures, and validation checks designed to ensure that the data capture system operates accurately, reliably, and consistently. With EDC, the designers of the structure may impose rules to maintain data and study quality, ranging from the simple (disallowing, for instance, data falsely labeled with future dates or biological values incompatible with life) to the more complex (for instance, implementation of automated checks of inclusion and exclusion criteria so that ineligible individuals are not enrolled as participants.

6.2.3.3 Data Security Procedures

Procedures for maintenance of data security – incorporating both prevention of deletion and prevention of theft – should follow and expand upon the Data Safety Monitoring Plan (DSMP) of the study protocol. Details on protocol development are provided in the Essential Documentations Module.

These procedures should delineate pertinent risks as well as the plan for prevention of theft or loss, including identification of the individuals responsible for maintaining and routinely confirming data security.

6.2.3.4 Data cleaning and quality control procedures

These describe the programmed and/or built-in procedures of the EDC system and CDMS ensuring data validity. Again, automated procedures (e.g. restriction of allowable values to those that match variable definitions) may be employed.

6.2.3.5 Data import and exports Procedures

These detail the frequency and methods that are to be used for the loading of data generated outside the EDC system (for instance as measured by a third party laboratory that provides data in a flat file external to the main trial database). Security and preservation of data quality must be accounted for here.

6.2.3.6 Case Report Forms

Case Report Forms (CRF) are the ‘source’ documents – which may be electronic forms - for recording the protocol-required information to be reported to the trial’s sponsor on each trial subject.

Paper or electronic documents should be consistent with the source documents (NIH_protocol_template). Investigator must warrant that the case reports be accurate, complete and legible.

Edits and corrections are to be documented by study personnel with initials and dates, and be available for audits. The corrections should not obscure, however, the original data entry such as the audit trail for either written or electronic data corrections.

At IFAR, the design, analysis, and preparation of the interim and final clinical trial reports should be done with the guidance of the Research Informatics and Biostatistics cores as appropriate.

6.3 Monitoring of Study Participants

Participant monitoring is a critical task for trial personnel. The CDMS should support and facilitate this task. Some exemplary functions are described below.

6.3.1 Screening/Recruitment

The EDC and CDMS should reinforce the inclusion/exclusion criteria such that only eligible participants are enrolled.

6.3.2 Randomization/Blinding procedures

The EDC should facilitate assignment of participants to treatments and control regimes in a transparent and error-free manner, while facilitating maintenance of blinding of study personnel as appropriate. See Experimental Design and Statistical Considerations Module.

6.3.3 Cessation of intervention

The CDMS should provide for notation and monitoring of cessation of intervention or other trial procedures as indicated by safety monitoring. Reason for and duration of cessation should be provided, along with other details Note that this is distinct from withdrawal, as described above.

6.3.4 Withdrawals

The CDMS should facilitate the recording of participant withdrawals from the trial. It should capture the reasons for withdrawal, relevant dates and other details. It should also facilitate any additional contact with participants and recording of data relevant to said contacts.

6.3.4.1 Tracking

The CDMS should facilitate tracking of participants’ enrollment and progress through the trial consistent with NIH and ICH guidance and with the reporting paradigm laid out in the Consolidated Standards of Reporting Trials (CONSORT) Statement. CONSORT is a set of recommendations for reporting the information for randomized trials including its design, methods, results, and conclusions (Consort). It comprises a 25-item checklist and flow diagram providing guidance for study enrollment, retention, and monitoring as well as derivation of analytic data sets and other data structures.

Figure 6.1 Study flow diagram

6.4 Reporting

A basic function of the CDMS is reporting of data and metadata relevant to trial monitoring. Prior to the trial startup the CDM should outline and describe the list of anticipated study reports. Some considerations are:

6.4.1 Safety

Safety reports should comply with the applicable regulatory requirement(s) and with the ICH Guidance for Clinical Safety Data Management (FDA_CDER_CBER: 5.17.2). The parties that must receive the safety reports include all concerned investigator(s)/institutions(s), the IRB(s)/IEC(s) where required, and regulatory authorities. See Human Subjects and Essential Documents modules for detail.

6.4.1.1 Adverse Events and Serious Adverse Events.

The CDMS should facilitate reporting of AEs and SAEs consistent with the study protocol and regulatory requirements (ICH_E2A). Consideration should be given to coding of event expectedness, severity and other relevant qualifications. See Human Subjects and Essential Documents modules.

6.4.2 Screening and Enrollment

The EDC and CDMS should be constructed to facilitate regular reporting of study progress, and specifically screening, enrollment and participant visit tracking. A reasonable model for said reports is contained in the DSMB reporting templates provided as part of the NIA Toolbox.

6.4.3 Data quality and completeness

The EDC and CDSM should be structured to facilitate production of data quality reporting to insure data accuracy, protocol compliance and adherence to regulatory requirements.

6.4.4 Progress and final reports

Most federally funded projects require yearly progress and final project reports. State usually done by the PI and grants administrator as they require scientific and administrative progress. Again, data management systems should be structured to interact with statistical software and reporting to facilitate production of these reports.

6.4.5 Additional reports

Project-specific reporting is par for the course in any clinical trial, and will reflect specific complexities of each design. Complex interventions, cluster designs, and numerous other features will necessitate redesign of standard reports or construction of new ones. The EDC and CDMS should be structured to permit easy access to study data and metadata for authoring of said reports within the CDMS or parallel technology (e.g. statistical analysis software).

6.5 Trial Documents and Data Retention

6.5.1 Retention of trial documents.

Source data should be attributable, legible, contemporaneous, original, accurate, and complete. Changes to source data should be traceable, should not obscure the original entry and should be explained if necessary (e.g., via an audit trail).

The ICH E6 guidance (FDA_CDER_CBER) recommends that trial documents be kept by the investigator/institution as specified in Essential Documents for the Conduct of a Clinical Trial and as required by the applicable regulatory requirement(s). See Essential Documents module.

6.5.2 Data Use Agreements (DUA)

SEE Data Use Agreements in the Human Subjects and Regulatory module (Also, IFAR_DUA, IFAR_sensitive_data_sharing).

6.6 Special Considerations for Older Subjects

Section 164.514(a) of the Health Insurance Portability and Accountability Act of 1996 (HIPAA) Privacy Rule provides the standard for de-identification of protected health information, including that of older subjects. Under this standard, health information is not individually identifiable if it does not identify an individual and if the covered entity has no reasonable basis to believe it can be used to identify an individual. For older adults over 89 years of age, states that “all elements of dates (including year) indicative of such age, except that such ages and elements may be aggregated into a single category of age 90 or older” (HHS_hipaa).

As noted above, the NIA Toolbox is an excellent resource for guidance of specific relevance to federal research in aging.

6.7 Appendix I

6.7.1 Network resources

At IFAR servers and network resources are accessible to approved staff only. The list of IFAR preapproved network resources includes:

  • IFAR file servers (e.g. IFAR NAS)
  • Application servers (e.g. REDCap, SharePoint, Accellion)
  • Database servers (e.g. MySQL, SQL Server)
  • Application Security and User Federation
  • IFAR Extranet and User Repository (LDAP)

According to IFAR’s sensitive data security policy projects are encouraged to maintain a dedicated network share on IFAR or HSL controlled file servers to ensure secure isolation of data and materials. Investigators must also follow HSL/IFAR procedures to request employee access to network resources (see IFAR_sensitive_data_security).

6.7.2 Systems available at IFAR

6.7.2.1 Electronic data capture (EDC)

Research Electronic Data Capture (REDCap) is a web application for building and managing online surveys and databases via secure web technologies. Each individual project in RedCap is provided a separate workspace and the data is stored in a MySQL relational database on the private corporate network behind several firewalls and located within the HSL data center. The system provides features like audit trails for tracking data entries and user activity, calendars for scheduling events, branching logic, file uploading and calculated fields (REDCap).

Open Clinica is an open source clinical trial software for EDC and CDM. The software is used to create project databases, develop eCRFs, create validation check, schedule events, capture eCRF data from study sites, monitor and manage the clinical data.

6.7.3 Data Analytics

STATA is a statistical software package used for data analysis, data management, graphics, simulations and custom programming for multiple types of data (Stata). The package is available for Windows, OS, X, and Linux. Its file format permits the exchange of data sets between users of different operating systems (iu_kb).

Statistical Analysis System (SAS) is a software package used to enter, retrieve, and manage data. It can also retrieve data for different operating systems to perform advance statistical and mathematical analyses, graphics and reporting (sas).

Statistical Package for the Social Sciences (SPSS) or IMB SPSS Statistics is a software used for statistical analysis, data mining, text analytics, and data collection. Like SAS can also retrieve data from different operating systems to perform statistical analyses, reports, and graphics.

R is a language and environment for statistical computing. Its suite of software facilities are used for data manipulation, calculation and graphic reporting. R has the advantage to allow users to add new programing functions.

6.7.4 Reporting systems

TIBCO Jaspersoft is an open source commercial software used in conjunction with other open source infrastructure like MySQL and JBoss to retrieve data, and develop reports (Swenson_2002). The JasperReports provides resports and analytics that can be embedded into a web or mobile application in a variety of formats and in real-time (Jaspersoft).

Shiny by RStudio is a web application framework for R that transforms analyses into interactive web applications (Shiny). The application allows the manipulation of the data by sorting, filtering, and by empowering the user to customize their analysis. At IFAR, Shiny is used to display different project consort and compliance reports in real time.

JIRA is an Atlassian software development tool used by agile teams to plan, track, release and report Issue tracking / tasks ticketing system.

Google Docs is a web-based application used to create, edit and store documents and spreadsheets. Is part of a package offered by and associated with Google Apps for Business and is compatible with most word processor applications and presentation software (Google_Docs).

Atlassian Confluence is a team collaboration software and enterprise wiki tool. It is used to create, organize, and discuss the contents of projects and/or knowledge base of different IFAR centers. The advantage of Confluence is that documents are edited in real time.

6.8 References

@article{Krishnankutty_2012, doi = {10.4103/0253-7613.93842}, url = {http://dx.doi.org/10.4103/0253-7613.93842}, year = 2012, publisher = {Medknow}, volume = {44}, number = {2}, pages = {168-172}, author = {Binny Krishnankutty and BR Naveen Kumar and LathaS Moodahadu and Shantala Bellary}, title = {Data management in clinical research: An overview}, journal = {Indian Journal of Pharmacology}, } @article{ Gerritsen_1993, doi = {}, PMID:{8130557} PMCID: {PMC2850658} url = { https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2850658/pdf/procascamc00002-0670.pdf}, year =1993, pages = {659–62}, author = {Gerritsen MG, Sartorius OE, vd Veen FM, Meester GT.}, title = {Data management in multi-center clinical trials and the role of a nation-wide computer network A 5 year evaluation}, journal = {Proc Annu Symp Comput Appl Med Care}, }

@website{NIH_standars_CR, title = {Standards for Clinical Research within the NIH Intramural Research Program}, organization = {National Intitutes of Health, Clinical Center}, year = {2009} url = {http://www.cc.nih.gov/ccc/clinicalresearch/standards1.html#info} }

@website{E6_R2), title = {Integrated Addendum to ICH E6(R1): Guideline for Good Clinical Practice E6(R2). Current Step 2 version}, organization = {International Conference on Harmonisation of Technical Requirements For Registration of Pharmaceuticals for Human Use}, year = {2015}, month = {June}, url = {http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R2__Addendum_Step2.pdf} }

@website{ICH_E2A, title = {ICH E2A Guidelines – Safety Data Management: Definitions and standards for expedited reporting}, organization = {International Council of Harmonisation} year = {1995}, month = {June}, url = {http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500002749.pdf}, }

@website{NIH_protocol_template, title = {Clinical Trials Protocol template – Office of Science Policy}, organization = {National Intitutes of Health - Office of Science Policy}, month = {February} day = {5} year = {2016} url = {http://osp.od.nih.gov/sites/default/files/Protocol_Template_05Feb2016_508.pdf} }

@website{IFAR_sensitive_data_sharing, title = {Institute for Aging Research Policies and Procedures - Sensitive Data Sharing Policy}, organization = {Institute for Aging Research} Location = {Roslindale, MA} url = {http://thehslhub/~/media/HSLNet/P_P/IFAR/IFARPPSensitiveDataSharing.ashx}, }

@website{IFAR_sensitive_data_security, title = {Institute for Aging Research Policies and Procedures - Sensitive Data Security Policy}, organization = {Institute for Aging Research} Location = {Roslindale, MA} url = {http://thehslhub/~/media/HSLNet/P_P/IFAR/IFARPPSensitiveDataSecurity.ashx}, }

@website{IFAR_sensitive_data_retention, title = {Institute for Aging Research Policies and Procedures - Sensitive Data Retention and Destruction Policy}, organization = {Institute for Aging Research} Location = {Roslindale, MA} url = {http://thehslhub/~/media/HSLNet/P_P/IFAR/IFARPPSensitiveDataRetentionandDestruction.ashx}, }

@website{IFAR_DUA, title = {Institute for Aging Research Policies and Procedures - Limited Data Sets and Data Use Agreements}, organization = {Institute for Aging Research} Location = {Roslindale, MA} url = {http://thehslhub/~/media/HSLNet/P_P/IFAR/IFARPPLimitedDataSetsandDataUseAgreements.ashx}, }

@website{IFAR_data_encrypt, title = {Institute for Aging Research Policies and Procedures - Data Encryption}, organization = {Institute for Aging Research} Location = {Roslindale, MA} url = {http://thehslhub/~/media/HSLNet/P_P/IFAR/IFARPPDataEncryption.ashx}, }

@website{IFAR_data_ret_destr, title = {Institute for Aging Research Policies and Procedures - Data Retention and Destruction}, organization = {Institute for Aging Research} Location = {Roslindale, MA} url = {http://thehslhub/~/media/HSLNet/P_P/IFAR/IFARPPSensitiveDataRetentionandDestruction.ashx}, }

@Website{ITHS_checklist, title = {Study Start-up checklist},
organization = {Institute of Translational Health Sciences}, address = {Seattle, WA} url = {https://www.iths.org/investigators/forms-templates/study-document-templates/}, grant = {UL1TR000423} }

@Website{FDA-study subjects, title = {Recruiting Study Subjects – Information Sheet}, organization = {U.S. Food and Drug Administration. U.S. Department of Health and Human Services}, address = {}, url = {http://www.fda.gov/RegulatoryInformation/Guidances/ucm126428.htm} }

@website{NIA-Guide MOP, title = {Guidelines for Developing a Manual of Operations and Procedures (MOP), Version 2}, organization = {National Institute on Aging}, year = {2011}, month = {September}, day = {30}, url = {http://webcache.googleusercontent.com/search?q=cache:LN9CyBbTsN4J:https://www.nia.nih.gov/sites/default/files/manualofproceduresmopfinal2_0.doc+&cd=1&hl=en&ct=clnk&gl=us} }

@webpage{FDA_CDER_CBER, title = {Guidance for Industry E6 Good Clinical Practice: Consolidated Guidance}, organization = {U.S. Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER) Center for Biologics Evaluation and Research (CBER)}, year = {1996}, month = {April}, url = {http://www.fda.gov/downloads/Drugs/…/Guidances/ucm073122.pdf} }

@book{Domansky_2009, title = {Successful Randomized Trials, A handbook for the 21st Century}, author = {{Domansky, M and S McKinlay}}, publisher = {Lippincott Williams & Wilkins}, address = {Philadelphia, PA}, year = 2009 }

@website{CFR21_part11, url = {https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11}, title = {The United States Code of Federal Regulations 21 {CFR}, Part 211}, }

@website{Health_Catalyst, title = {Clinical Data Management: 3 Improvements Startegies}, url = {https://www.healthcatalyst.com/improve-clinical-data-management-healthcare-reduce-waste?gclid=CLTestHd-ssCFUYehgodqzoO8Q} }

@website{Consort, title = {Consort transparent Reporting of trials}, organization = {Consort}, url = {http://www.consort-statement.org/consort-statement/flow-diagram} }

@website{Harvard_Catalyst, title = {The Harvard Clinical and Translational Science Center. An Investigators’ Guide to Research Data Management Practices}, organization = {Harvard Catalyst}, url = {https://catalyst.harvard.edu/pdf/regulatory/Investigators%20Guide%20to%20RDM%20practice.pdf} }

@website{UWA_toolkit, title = {Research Data Management Toolkit: Research Lifecycle}, organization = {The University of Western Australia}, url = {http://guides.is.uwa.edu.au/content.php?pid=319161&sid=2616069} }

@website{nidcr_toolkit, title = {Toolkit for Clinical Researchers}, organization = {National Institute of Dental and Craniofacial Research}, url = {http://www.nidcr.nih.gov/research/toolkit/} }

@website{hipaa_deid title = {Health Information privacy: The De-identification Standard}, organization = {U.S. Department of Health & Human Services}, url = {http://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html} }

@article{Krishnankutty_2012, doi = {10.4103/0253-7613.93842}, url = {http://dx.doi.org/10.4103/0253-7613.93842}, year = 2012, publisher = {Medknow}, volume = {44}, number = {2}, pages = {168}, author = {Binny Krishnankutty and BR Naveen Kumar and LathaS Moodahadu and Shantala Bellary}, title = {Data management in clinical research: An overview}, journal = {Indian Journal of Pharmacology}

@website{Clinical_Tools, title = {Guidelines for Responsible Data Management in Scientific Research}, organization = {Clinical Tools Inc} Funded = {Office of Research Integrity, US Department of Health and Human Services} url = {http://ori.hhs.gov/images/ddblock/data.pdf}, }

@article{Gallagher_2014, doi = {10.1016/j.urolonc.2012.11.019}, url = {http://dx.doi.org/10.1016/j.urolonc.2012.11.019}, year = 2014, month = {jan}, publisher = {Elsevier {BV}}, volume = {32}, number = {1}, pages = {32.e1–32.e9}, author = {Sarah A. Gallagher and Angela B. Smith and Jonathan E. Matthews and Clarence W. Potter and Michael E. Woods and Mathew Raynor and Eric M. Wallen and W. Kimryn Rathmell and Young E. Whang and William Y. Kim and Paul A. Godley and Ronald C. Chen and Andrew Wang and Chaochen You and Daniel A. Barocas and Raj S. Pruthi and Matthew E. Nielsen and Matthew I. Milowsky}, title = {Roadmap for the development of the University of North Carolina at Chapel Hill Genitourinary {OncoLogy} Database{}{UNC} {GOLD}}, journal = {Urologic Oncology: Seminars and Original Investigations} }

@website{HHS_hipaa, tittle = {Health Information privacy: The De-identification Standard}, organization = {U.S. Department of Health & Human Services}, url = {http://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html}, }

@website{Catalyst, title = {Harvard Catalyst Regulatory Atlas} organization = {The Harvard Clinical and Translational Science Center}, url = {http://connects.catalyst.harvard.edu/regulatoryatlas/?mode=c&id=51} }

@website{BCH_RPG, title = {Research Practive Guidelines} organization = {Boston Children’s Hospital}, url = {http://www.childrenshospital.org/research-and-innovation/research/clinical/clinical-research-center/research-practice-guidelines} }

@website{BCH_RPG-05, title = {Guideline for Developing a Manual of Operations (MOO)-RPG-05} organization = {Boston Children’s Hospital}, url = {http://www.childrenshospital.org/~/media/research-and-innovation/clinical-research-center/rpg_03_final-12115/rpg052014.ashx?la=en} }

@website{BCH_RPG-07, title = {Guidelines for Research Data Collection-RPG-07} organization = {Boston Children’s Hospital}, url = {http://www.childrenshospital.org/~/media/research-and-innovation/clinical-research-center/rpg0711-14-14final.ashx?la=en} }

@website{NIA_NIH_toolbox, title = {Clinical Research Study Investigator’s Toolbox} organization = {National Institute on Aging}, url = {https://www.nia.nih.gov/research/dgcg/clinical-research-study-investigators-toolbox/startup} }

@website{OpenClinica, title = {Fundamentals Aspects of Electronic Data Capture} organization = {Open Clinica}, url = {https://www.openclinica.com/electronic_data_capture/} }

@website{REDCap, title = {REDCap} organization = {REDCap }, url = {https://projectredcap.org} }

@website{Stata, title = {Stata} organization = {Stata}, url = {http://www.stata.com/) }

@website{iu_kb, title = {Indiana University Knowledge base} organization = {Indiana University}, url = {https://kb.iu.edu/d/afly} }

@website{sas, title = {SAS the Power to Know, Introduction to the SASA System} organization = {SAS}, url = {http://support.sas.com/documentation/cdl/en/basess/58133/HTML/default/viewer.htm#a002645411.htm} }

@website{R, title = {What is R} organization = {R}, url = {https://www.r-project.org/about.html} }

@article{Swenson_2002, url = {http://www.javaworld.com/article/2074594/java-security/reports-made-easy-with-jasperreports.html}, title = {Reports made easy with JasperReports, The open source JasperReports uses XML templates for your reporting needs} publisher = {JavaWorld}, }

@website{Jaspersoft, title = {JasperReports ® Server} organization = {Jaspersoft Community}, url = {http://community.jaspersoft.com/project/jasperreports-server} }

@website{Shiny, title = {Shiny by RStudio} organization = {Rstudio}, url = {http://shiny.rstudio.com/} }

@website{Google_Docs, title = {Google Docs} organization = {WhatIs.com}, url = {http://whatis.techtarget.com/definition/Google-Docs} }

@website{Confluence, title = {Confluence} organization = {Atlassian Confluence}, url = { https://www.atlassian.com/software/confluence} }