Data Continuity and Data Integrity
This chapter will cover:
- Definition of Data Continuity and Data Integrity
- Ways to ensure Data Continuity and Data Integrity
Digital continuity refers to the ability to use data in the way in which it is needed for as long as it is required. Maintaining digital continuity requires data to be actively managed throughout its lifetime to ensure its integrity.
Data integrity is the degree to which data are complete, consistent, accurate, trustworthy, reliable, and these characteristics of the data are maintained throughout the data lifecycle. In order to preserve the integrity of data, it’s necessary to establish mechanisms that will seek to maximise compliance with ALCOA+ principles.
For data in Laboratory Information Management Systems, electronic notebooks, lab equipment, and applications (to name a few), maintaining continuity and integrity can prove challenging because the data generated and held in these systems is often in proprietary formats. These formats are dependent on specialist software, hardware, and operating systems to be accessed, read, and understood.
It’s worth noting that some of these data formats will be more durable and stable over time than others.
Although there are no specific mandates regarding the location or format of archived lab data, that data must remain accessible and retrievable in human-readable form throughout the retention period.
"To ensure digital continuity and data integrity, it’s vital that the archivist liaises with the data owner, test facility management, quality assurance, information technology, and digital preservation solution vendor."
To ensure digital continuity and data integrity, it’s vital that the archivist liaises with the data owner, test facility management, quality assurance, information technology, and digital preservation solution vendor, to:
- Plan methodically.
- Determine the longevity of native and proprietary data formats (although this can be challenging to know in advance).
- Address dependencies on software, hardware, and operating systems.
- Assess the risks presented and establish effective processes to control any necessary migration and /or conversion to a preservation format.
- Consider the metadata necessary to ensure referential integrity and relevant to maintaining a thorough understanding of the data.
Sometimes it is possible to generate summary data in PDF format and/or export that summary data to a structured format such as CSV or XML.
However, while PDF and XLSX may provide an alternative format for archiving, converting to PDF may lead to a loss of some underlying data and XLSX permits further processing and modification, which regulations prohibit..