Out of Lotus Notes – Our Experiences
In many companies, older Lotus Notes applications are not fully shut down even after having been replaced by new applications. The uncertainty about how to handle the information stored in the application holds the companies back from taking the final step. This leads to compliance challenges as well as licensing costs. In this article, we describe some of our experiences with shutting down Lotus Notes.
Domino and Lotus Notes are a technology and a platform that have served – and still serve – many companies well. Nevertheless, in recent years, many companies have replaced Lotus Notes applications with other platforms and applications. However, the final phase-out of Domino/Lotus Notes is complicated by the fact that the old applications are not fully shut down but kept alive to preserve the information stored in them.
Given that the Domino/Lotus Notes platform is highly flexible, companies have built applications for many purposes and often in large numbers. As a result, the applications are widely different and often not well-documented. The information stored in the applications is therefore organized in widely different ways and often involves very large, inhomogeneous quantities. The large quantities, the diversity, and the lack of documentation all complicate the work of managing the amount of information and extracting it in a suitable manner.
Lotus Notes possesses some technical features that make the platform particularly flexible and widely applicable. However, these same features make it complicated to manage the information stored in the application. In the following, two of these features are described: Lotus Notes documents and Lotus Notes’ view structure.
Lotus Notes Documents
Most people associate the word “document” with a file containing information. It could be a Word file, a PDF file, a spreadsheet file, an image file, etc. However, a Lotus Notes document is not a file, but rather resembles an email in many ways. Like an email, a Lotus Notes document has a body field with text and the ability to attach files.
The text in the body field itself is information that typically needs to be preserved. The text does not come from a file that is opened and displayed by an application, as Word opens and displays a Word file. Rather, the text comes from a Rich Text field in the Lotus Notes database. To preserve the text elsewhere outside the database, it must be saved as a file that shows exactly what would have been seen if the Lotus Notes database still existed and displayed this information.
There are different approaches to solving this problem, and a commonly used method is to create PDF files using technology that is leased or purchased for this purpose. A conversion, as this is, challenges the integrity of the document, as it is the content itself that is being processed. The conversion must therefore take place in a controlled and documented process. (Read more about the integrity of documents here).
The text from the body field in Lotus Notes can be preserved by creating PDF files in a conversion that takes place in a controlled and documented process.
Any attached files are, of course, already files, however, in their case, a conversion to PDF may be relevant if they are in a format that does not have durability equivalent to the horizon for which the files should be preserved.
Here, it is assumed at a very basic level that PDF is a good choice. It often is. However, PDF comes in many types and versions, and in each individual situation, it must be carefully considered which type and version is best suited. Sometimes PDF is not the right choice. Put very simply, the chosen format must be durable for as long as the document needs to be preserved. (This involves so many significant considerations that it must be addressed in a separate article.)
Attached files also pose another type of challenge, as it is often relevant to preserve the files’ relationship to the text to which they were attached. Strator has contributed to solving this for both emails and Lotus Notes documents. It has been solved in various ways depending on how the documents would be used in the future, where they would be stored, and what functionality for specifying relationships might be available.
Lotus Notes Application View Structure
Many Lotus Notes applications are commonly referred to as “databases” in businesses, and for good reason. Simplistically, many Lotus Notes applications are just different views of data from the Lotus Notes database.
These views, as they are called in database language, are essentially a well-formatted display of the results of a search based on specified criteria. The view displays the currently existing documents that meet the criteria. These criteria are based on metadata about the documents in the Lotus Notes database.
This means that Lotus Notes applications, technically described, are a well-organized collection of documents with metadata that are intricately linked to views so that the different views display subsets of the documents with varying business relevance.
Technically described, Lotus Notes applications are a well-organized collection of documents with metadata that are intricately linked to views so that the different views display subsets of the documents with varying business relevance.
Most document management and archiving systems have an architecture in which each subset of documents is assigned a document type that determines the set of metadata associated with the document. In principle, all of this metadata is relevant to the document and its display and other use in the system.
However, the Lotus Notes applications that Strator has helped decommission have been fundamentally different in that all metadata fields exist on all documents. Whether the fields are filled out and used depends on which views they are included in. In practice, metadata that represents the same business information may appear in two different fields in two different views (for example, one field may be called “Month” with a value of “July,” while another field may be called “Month Number” with a value of “7,” but both represent the same month).
Because the use of the application typically changes over time and new views and metadata are added later (or due to poor architecture from the start), inconsistencies in this metadata, stemming from inconsistencies in the application’s structure, can often be found. (In the example above, a document may have “July” as the month, but “4” or no value in the Month Number field.)
If the various metadata and the logic in views are well-documented, it is relatively easy to gain insight into the information in the application and identify the central views and metadata for each subset of documents. If this is lacking, it is fortunately possible to analyse the configuration of the application and derive key parts of the logic, but as explained above, the analysis can be complex and leave some questions unanswered.
The worst thing that can happen is that documents are lost when an application is shut down. Therefore, it is an absolute requirement that all documents are included (or that a conscious decision has been made to exclude certain ones). The danger with this type of task is always that you know what you see – but you don’t know what you don’t see.
The danger of extracting documents during shutdown is always that you know what you see – but you don’t know what you don’t see.
The different views display subsets of the documents. As already mentioned above, the criteria in a view are decisive for which documents are displayed. Additionally, access restrictions will also affect whether a document is displayed or not.
To ensure that all documents in the database are considered, it is therefore absolutely necessary to get “behind” both views and access restrictions. What can be seen must be compared with what is displayed in the different views. There may be documents with inconsistent metadata, as in the example above, and there may also be documents that are not displayed in any view and are therefore completely invisible to users. It may be relevant to discard these, but it must not be done in ignorance of their existence.
The structure of Lotus Notes applications can thus make it somewhat complex to get an overview of the content and which metadata is central to each document. According to Strator’s experience, this can be done relatively painlessly by analyzing thoroughly and approaching it in a very structured way.
Clean-Up and Archiving
Applications that have had a long life, such as the typical Lotus Notes application, often have a large and inhomogeneous set of documents. It should always be considered where, for how long, and in which format documents should be stored. Some of these considerations have already been touched upon above.
However, it is also essential to consider what should be saved. It is possible to extract and save everything, but there are situations where it is better to do a more or less thorough clean up.
It may be necessary to transfer some of the documents to a functioning system where they will play an active role for users. For these documents, it is valuable to clean up thoroughly and assign good metadata in connection with the migration to the new system.
If the documents are to be archived outside of active use, it should be considered whether there are documents that may not be kept (e.g., due to GDPR), and if so, an effort must be made to identify them so they can be discarded. Additionally, factors such as overview of retention time and retrieval can determine which clean-up effort is appropriate.
Out of Lotus Notes
It happens that legacy applications are not shut down completely, even though they are no longer in use. This seems to be a particularly widespread phenomenon for applications on the Domino/Lotus Notes platform. In this article, we have touched on some particular issues that apply to Lotus Notes applications – both the document concept and the view structure. At Strator, we believe that these can be contributing factors as they complicate the extraction and continued preservation of documents.
However, if you are aware of and address these issues, it is perfectly possible to completely responsibly and with a reasonable effort shut down the platform, thereby overcoming compliance challenges and eliminating licensing costs.
Good luck with your work.
