Technology Assisted Review (TAR) is a marketing term used in the eDiscovery community to describe the process of automatic classification of documents in a so-called legal review. Similar documents are classified based on training data or seed sets. Typical classes include Confidential, Privileged or Responsive. As the saying goes, “there’s more than one way to skin a cat”; TAR is also called Machine Assisted Review (MAR), Computer Assisted Review (CAR), Predictive Coding, Concept Search, or Meaning-based computing: all of which are marketing terms without any specific scientific meaning.

A recent US ruling by Judge Peck regarding the use of machine learning technology in legal review, has created a lot of tumult in the eDiscovery community (see http://www.law.com/jsp/lawtechnologynews/PubArticleLTN.jsp?id=1202542221714 for more information and links to other articles). Not only has the opposing party filed complaints against the ruling (including accusing the judge of obtaining financial benefits from the vendor whose software was used), but the entire legal community seems to be engaged in the heated topic.

Now that there is case law on the use of TAR, and it has been confirmed by other judges, one can expect a dramatic increase in Predictive Coding, Concept Search or other terms relating to TAR capabilities being a requirement for eDiscovery software buyers.

The Science Behind TAR

For myself, I am an avid fan of artificial intelligence and machine learning. As a matter of fact, I have practiced in the field since 1985 and hold the special chair of text-mining at the University of Maastricht, where I teach my students everything there is to learn on the application of text mining for applications such as document classification, clustering and information extraction.

Machine learning and other techniques from artificial intelligence are not based on “hocus pocus”: they are based on solid mathematical and statistical frameworks in combination with common-sense or biology-inspired heuristics. In the case of text-mining, there is an extra complication: the content of textual documents has to be translated, so to speak, into numbers (probabilities, mathematical notions such as vectors, etc.) that machine learning algorithms can interpret. The choices that are made during this translation can highly influence the results of the machine learning algorithms.

For instance, the “bag-of-words” approach used by some products has several limitations that may result in having completely different documents ending up in the exact same vector for machine learning and having documents with the same meaning ending up as completely different vectors. See http://www.aiim.org/community/blogs/expert/Language-is-Not-Just-a-Jumbled-Bag-of-Words-Why-Natural-Language-Processing-Makes-a-Difference-in-Content-Analytics for more information on this topic. The garbage-in, garbage-out principle definitely applies here!

Other complications arise when:

• More than one foreign language is used in the document set, for instance, if some documents are in English and some documents are in Dutch. Multi-lingual documents in which multiple languages appear in individual documents causes even more problems.
• The more document categories there are, the lower the quality will be for the document classification. This is very logical as it is easier to differentiate only black from whitethan it is to differentiate 1,000 types of gray values.
• The absence of sufficient relevant training documents will lower the quality of classification. The number of required training documents grows faster than the increase of the number of categorization classes. So, for 2 times more classes one may need 4 times more training documents.
• The documents use very different or very ambiguous language for the same topics (e.g. there are many synonyms and homonyms).

Dealing with incremental document collections (e.g. new documents are added after training) will result in lower quality or require completely new training of the machine learning.

Several risk factors are listed here, but there are more depending on the specific machine learning technology that is used: technology that is based on Bayes classifiers (falsely) presumes statistical independence between measured features (e.g. word occurrences) and Latent Semantic Indexing (LSI) and its variants such as Probabilistic Latent Semantic Analysis (PLSA) effectively use a lossy information compression algorithm (SVD) that may result in more (irreversible) information loss than required. Knowledge of the specific parameter settings is integral to gaining a full understanding of the quality of specific machine learning models.

There is No Free Lunch

Machine-learning requires significant set-up involving training and testing the quality of the classification model (aka the classifier) , which is a time consuming and demanding task that requires at least the manual tagging and evaluation of both the training and the test set by more than one party (in order to prevent biased opinions). Testing has to be done according to best practice standards used in the information retrieval community (e.g. see the proceedings of the TREC conferences organized by the NIST). Deviation from such standards will be challenged in courts. This is time consuming and expensive and should be factored into the cost-benefit analysis for the approach.

If the classifier does not work (e.g. a mutually-agreed upon predefined quality level is not reached), only retraining the entire model with better training examples will work. Eventually, this process could negate any performance increases or cost savings that could have been achieved by applying the technology. In that event it is impossible to improve the model and all training and test efforts will have been a waste of time. This may very well happen in cases that suffer from the complications as described above.

Additionally, one has to be able to explain and defend the application of machine learning technology in court. This may not be a trivial task given the fact that machine learning is based on state-of-the-art principles in linear algebra and probability calculus that are not commonly understood by those who may be involved in the law suit. Therefore, parties and the court will rely heavily on (expensive) expert witnesses.

Summary

So, when applying Predictive Coding, Concept Search or other names that refer to Technology Assisted Review, first become informed of the potential risks of using the technology for a particular case. It may be the right choice for some cases, but not for others.

If it is not possible or too risky to apply machine learning techniques, then there are also other forms of automatic document classification, such as rules-based document classification. These may be a better choice to use in certain cases, especially when defensibility is an issue. They come with their fair share of set-up, but in almost all cases they are more defensible and easier to manage.

At one point or another in the course of business, many companies will engage in litigation or have to answer to regulators. One common thread between companies is that the process of responding to litigation and regulatory inquiries can derail their normal business processes and involve unnecessary costs.  Fortunately, there are ways in which companies can become better prepared for inquiries and disclosures and even reduce costs.

Mark Mack, ZyLAB’s Enterprise Technology Counsel, discusses examples of achieving eDiscovery cost savings with the Metropolitan Corporate Counsel, saying, “Automatic redaction, the reuse of work product, the random sampling and our search technology are the pillars of cost savings for our clients.” [1] Companies that partner with ZyLAB receive world-class eDiscovery technology and see firsthand how it provides immediate eDiscovery cost savings, but also litigation readiness and improved information governance.

Some key topics and excerpts from the article:

Da Silva Moore and Technology Assisted Review

When asked her thoughts on the case and its impact on eDiscovery, Mary said, “I support Judge Peck’s judicial management toward reducing costs, speeding trials and encouraging cooperation. Judge Peck is one of a handful of jurists who understands and can communicate about computers, and who is willing to help educate the bar.”

In reference to some predictive coding products: “with the seed set protocol, counsel can find themselves forced to disclose the seed set(s) of documents fed into the software –even the nonresponsive documents.”

According to Mary, the ZyLAB difference in these instances lies in, “Using the ZyLAB rules-based approach, it is much less likely our clients will need to disclose a sample set of nonresponsive documents.”

Looking forward Mary says, “I expect to see in the next year that a random sampling of what is left behind will be the most significant contribution to cost savings for corporations. This will allow them intelligently to exclude custodians, which is one of the most significant ways of reducing costs.”

Having Better Control Over All Data: Reduce Overall Costs

The article describes quantitative versus qualitative Early Case Assessments (ECA) and how “the legal team winds up with far more insight early on, which can reduce costs by accelerating the settlement process or reducing the scope.”

Additionally, Mary also touches on the importance of supporting design drawings like AutoCad: “Some may have a specific need to find relevant schematics. Others may need automatic redaction of personally identifiable information, such as Social Security numbers. This results in double savings because it is both a huge time saver as well as an hourly rate saver.”

With respect to data monitoring, Mary addresses the possibility for corporations to “proactively monitor sensitive documents to avoid data seepage and loss of trade secrets. This is an enormous cost savings and also helps protect the corporation in this era of the FCPA, UK Bribery Act, Dodd-Frank and whistleblowers.”

She also communicated the possibility of leveraging multiple eDiscovery models strategically: “From a cost perspective, it is important to find an eDiscovery company that can support such seamless and cost-effective transitions from processing services, to SaaS, to on-premise.”

Proactive eDiscovery

Reactive to eDiscovery, even with the best technology at your side, can involve potentially unnecessary costs and disruptions to operations. During this interview, Mary discusses how corporations are repurposing eDiscovery software like the ZyLAB eDiscovery & Production System: “This proactive eDiscovery tool leads to intelligent governance and litigation readiness for the long term and attacks costs at the source: too much data that has no business or legal purpose.”

To read the full interview with Mary Mack, and get access to all her valuable insights, visit the ZyLAB media coverage page here.

Coming up…..Don’t forget to download the upcoming whitepaper from George Socha concerning re-purposing eDiscovery software for proactive purposes and achieving a state of litigation readiness. To reserve a complimentary copy in advance, please visit this page.

European legal professionals are encouraged to join ZyLAB EU/UK’s teams for a webinar, “Cross-border Litigation: Preparing for the Unknown,” this Thursday, March 29th.

This webinar will be lead by e-Disclosure authority Chris Dale and information management expert Johannes Scholtes and will dive into the specifics of data privacy and how it impacts eDiscovery during cross-border litigation. This webinar is part of ZyLAB EU/UK’s continuing European series of discussions for legal professionals and eDiscovery experts to explore the various challenges and scenarios when working within foreign courts and overseas legal systems.

However, some text analytics products take a so-called “bag of word” (BOW) approach, in which all words (maybe with the exception of a list of high frequency noise words) are dumped into a mathematical model, without any additional knowledge or interpretation of linguistic patterns and properties such as word order (“a good book” versus “book a good”), synonyms, spelling and syntactical variations, co-references and pronouns resolution or negations.

This “bag of words” approach takes simplicity one step too far. Here is why.

There are special issues that one has to take into account when applying, for instance, entity-, fact-, event- and concept extraction techniques in text mining and where Natural Language Processing can make the difference:

• Variant Identification and Grouping: It is sometimes needed to recognize variant names as different forms of the same entity giving accurate entity counts as well as the location of all appearances of a given entity. For example, one needs to recognize that the word “Smith”, in the example, refers to the “Joe Smith” identified earlier and therefore groups them together as aliases of the same entity.
• Normalization: Normalizes entities such as dates, currencies, and measurements into standard formats, taking the guesswork out of the metadata creation, search, data mining, and link analysis processes. An example would be good here…
• Entity Boundary Detection: Will the technology consider “Mr. and Ms. John Jones” as  one or two entities? And what will the processor consider to be the start and end of an excerpt like,  “VP John M.P. Kaplan-Jones, Ph.D. M.D.”?

Such basic normalizations will not only dramatically reduce the size of the data set, it will also result in better data analysis and visualization: entities that would not be related without normalization can be the missing link between two datasets especially if they are written differently in different parts of the data set or if they are not recognized as being a singular or plural entity properly.

In addition, one of the other problems in the discovery and identification of entities, facts, events and concepts, is the resolving of the so called anaphora and co-references. This is the linguistic problem to associate pairs of linguistic expressions that refer to the same entities in the real world.

Consider the following text:

“A man walks to the station and tries to catch the train. His name is John Doe. Later he meets his colleague, who has just bought a card for the same train. They work together at the Rail Company as technical employees and they are going to a meeting with colleagues in New York.”

The text contains various references and co-references. Various anaphora and co-references will have to be disambiguated before it is possible to fully understand and extract the more complex patterns of events. The following list shows examples of these (mutual) references:

• Pronominal Anaphora: he, she, we, oneself, etc.
• Proper Name Co-reference: For example, multiple references to the same name.
• Apposition: the additional information given to an entity, such as “John Doe, the father of Peter Doe”.
• Predicate Nominative: the additional description given to an entity, for example “john Doe, who is the chairman of the soccer club”.
• Identical Sets: A number of reference sets referring to equivalent entities, such as “Giants”, “the best team”, and the “group of players” which all refer to the same group of people.

There are various ways of approaching these problems: (i) with an in-depth linguistic analysis of a sentence, or (ii) using a large already-annotated corpus. Both techniques have their advantages and disadvantages. There is still a lot of research required in this area in the coming years to improve the quality of these types of analyses, but there are already many reliable techniques to resolve co-references and anaphora.

Now, if all these natuaral language processing techniques are applied one will be able to:

• Extract 2-4x more (and better) entities, facts, events and concepts than other providers that use, for instance, bag of words technology which completely ignores linguistic patterns, negations, pronouns and co-references.
• Increase overall extraction quality easily with 20-30% up to 95+% recall, especially due to the proper interpretation of negations and better boundary detection.
• Provide a smaller and better set for data analysis such as the derivation of relationship networks and correlation patterns between custodians in social networks and email.
• Create much better training sets for machine learning algorithms such as those used for machine assisted review or predictive coding.

Language is not a jumbled bag of words; ignoring simple linguistic structures such as synonyms, spelling and syntax variations, co-references and pronouns resolution or negations will result in technology that will probably never exceed 60-65% recall and precision. This stunted performance is caused by the simple fact that lots of relevant information is ignored or wrongly used to build training sets for machine learning. To end users, 60-65% is only 15% off from random selection, which is often interpreted as unreliable behavior for eDiscovery, Governance, Enterprise Information Archiving and other Information Management initiatives. Yet surprisingly, many eDiscovery-related software products do not support Natural Language Processing, and therefore, they completely ignore relevant linguistic patterns like those described in here.

Of course, there are a few exceptions; some eDiscovery and information management products do offer Natural Language Processing as part of their text analysis.

For text mining, a very in-depth analysis is often not necessary; a reasonably light analysis can be sufficient to identify the most important elements of a sentence: the subject clause, the verb clause, potential proper nouns, references, and other relationships. In many cases finite-state parsers or shallow parsers can be used with the support of dictionaries. These analyses are also commonly known as a part-of-speech (POS) analysis.

This is when organizations discover the difference between world-class, end-to-end eDiscovery technology and other products on the market. Watch our new “Discover eDiscovery” video to see how ZyLAB’s powerful exploratory search is at the heart of our industry-leading software for eDiscovery, investigations and governance.

One component of a robust eDiscovery and information management platform is exploratory search. Unlike more traditional search tools, a legal or exploratory search product supports all the disparate data in the enterprise and retrieves all of the potentially relevant hits—not just the most popular ones. With the increased usage of cloud computing, social media, and multimedia, partnering with an eDiscovery software and services leader that handles every data format, source, and language with ease is crucial.

For example, the ZyLAB eDiscovery & Production System delivers advanced, exploratory searches across all electronically stored information in 400+ foreign languages. ZyLAB is the only eDiscovery and information management product that is optimized for 100% recall of relevant data through a combination of advanced search methods and the ability to interpret and handle complex files. Our technology reveals that crucial piece of evidence no matter how deeply it is buried in the organization. Plus, our software is proven, innovative, and fosters global best practices, such as Electronic Discovery Reference Model (EDRM) and Sedona.

So, when litigation—or the regulator—is knocking at your door, ZyLAB helps you save cost and time when there is none to spare.

Nowadays professionals carry out many business tasks remotely and via cloud-based tools. Plus, more and more courts are compelling production of “personal” data within online email. It is critical that those sources are properly supported by eDiscovery and information management software.

With the multitude of cloud-based email products and with potential evidence being stored in calendars, contacts, messages, attachments and a variety of other places, this new collector can help alleviate some of the obstacles of getting to the information. Are custodians using Gmail? Office 365*? Not a problem.

The new Cloud Collector pulls data from Microsoft Exchange Online, part of Office 365*, email messaging applications that support Exchange ActiveSync Protocol, IMAP and Post Office Protocol (POP3) making it much more possible to collect and search email data from the diverse systems used by custodians.

Being able to efficiently and effectively collect web-based email data and process it alongside all of the other ESI makes for a more complete picture of the case and potential evidence.

Examples of technology to support this approach are:

• Smart identification of custodians and data locations with automatic legal hold interviews. During the legal hold interview, software (like ZyLAB’s) can provide the customer with interview tools to determine a (very) early case assessment. Such interview questions are based on ABA best practices for ECA.
• Automatic and recurring collection of data, not only by location, file name, extension, and MIME type, but also by using full-text search to collect only relevant documents that match the negotiated Boolean.
• Exact de-duplication: NIST, in-house NIST, against other custodians, within custodian, and in production.
• Support in the review process for batch coding in the case of email threads, related documents, (near) duplicates, etc.
• Organizing review documents in groups with Machine Assisted Review technology to identify potentially privileged, confidential and responsive documents.
• Automatic redaction.

But there are many cases where a more qualitative approach can be very efficient and valuable to a $500/hour attorney. This is especially when negotiating a settlement—which happens in 97% of all cases. In this scenario, parallel to the quantitative approach described above, a team of attorneys (often mediators) searches and analyzes the data to look for strong evidence that will allow them to accelerate the negotiation and settlement process with the opposing party. All data that is ingested and processed is automatically enriched with advanced content analytics and is available for in-depth search and advanced analysis and visualization. Data can be mapped against the claims and relevant documents can be found quickly and easily. In addition, valuable insights gained from “the earliest early case assessment” help steer the eDiscovery strategy and budget, and minimize the legal risk, exposure and expenses from later stages.

Using this parallel approach, often a settlement is reached well before the quantitative data processing work is done and clients are saved a lot of money. There are more and more high-end law firms who prefer to make money providing valuable legal advice to their clients (for instance in a settlement process), which also has a much higher margin to them than a low-value and low-margin document processing and review services and which also results in happier (returning) clients.

Yes, I can see why Qualitative Early Case Assessment is giving Quantitative Early Case Assessment and run for the money!

But there was a new trend at the show this year. For the first time, visitors sought proactive information management, enterprise information archiving, legacy information clean-up, defensible disposal, data monitoring for internal investigations such as non-compliance, early fraud detection and other intelligent governance tools. The focus is shifting away from solely limiting risks and costs on specific matters, but looking to use the software for better knowledge management and to assist an organization to implement its strategic goals.

For most vendors, the marketing focus has remained squarely on  reactive eDiscovery solutions for the simple reason that these generated most revenue and closed many times faster than traditional Governance, Archival and other Information Management solutions.

But, we now see that organizations that have bought and deployed reactive eDiscovery solutions are now looking to leverage the same technology, people and skills for more proactive programs! This is especially true when the litigation work load is low, which is often the case, as litigation does not occur on a predictable timeline. There is really no excuse anymore not to use the tools and skills that one already has to limit risks and cost of risky custodians, locations and projects, let alone the benefits of less storage, less duplicates, less backup and better knowledge management.

So, finally, the only and ultimate solution to reduce eDiscovery risks and cost is being implemented: proactive eDiscovery, Governance and Information Archiving programs. All thanks to reactive eDiscovery!

There was no shortage of software solutions that could be deployed either in-house or in the cloud, but the availability of eDiscovery solutions that are fully parallel with distributed Virtual Machine architectures was minimal. I predict that will be the buzz next year, as the capability makes perpetual scalability easily attainable. Some of the showcased solutions cover the entire EDRM, whereas others address just parts of it. Some vendors also provided legal expertise to implement a solid legally-defensible methodology including templates, a well-documented quality control methodology, and referencable case law.

We saw the beginnings of new approaches to Early Case Assessment, where next to the traditional quantitative data-processing eDiscovery approach, some vendors also made it possible to use advanced text, audio and image search in combination with content analytics and data visualization for more qualitative Early Case Assessment (ECA) approach. We also saw advancements in Machine Assisted Review (including automatic creation of random data-sampling sets), Machine Translation and Automatic Redaction on hundreds of languages, electronic file formats and different database repositories including popular cloud-based data collections.

But, besides all these high-tech innovations, one of the key questions remains: can you collect from “my repository” – which is often comprised of legacy email archives or other obsolete content management systems for which support no longer exists. I hope that all the eDiscovery pain and challenges related to these proprietary archives makes organizations look for more enduring and sustainable open archive solutions in the future. In reality, if everything had been archived in XML, for instance, than collection would be a simple endeavor as well.

Another continuing question relates to legal production formats. Many law firms negotiate sometimes exotic or at least uncommon requests or they have found out that TIFF is not the best format to produce XLS (formulas), PPT (animations) or video. Intelligent and flexible legal production functionality can make or break a deal.

Also, for the first time at LegalTech, many visitors shied away from the eDiscovery bells and whistles and wanted to learn more about “proactive eDiscovery” for information management, enterprise information archiving, legacy information clean-up, defensible disposal, data monitoring for internal investigations such as non-compliance, early fraud detection and other intelligent governance tools. The focus seems to be shifting away from solely limiting risks and costs on specific matters, but looking to use the software for better knowledge management and to assist an organization to implement its strategic goals.

In the upcoming weeks, I plan to elaborate in more detail on these topics in my blogs, so look out for more!

During LegalTech New York, ZyLAB will showcase financial models for organizations to make informed decisions about eDiscovery in-house versus outsourcing and to pinpoint unnecessary over-spending on eDiscovery work.

Stop by booth 325 at LegalTech New York (#LTNY). All you need is a few minutes to see our state-of-the-art eDiscovery tools and calculate the impact they will have on your organization. See you there!

Or visit us at www.zylab.com.