Legal document similarity: a multi-criteria decision-making perspective

Identification of basic concept words

Natural Language Processing (NLP) offers a wide range of methods and approaches for the identification of topics from a document. Traditional TF-IDF based vector space model and Latent Dirichlet Allocation (LDA) use the distribution of words (Moody, 2016) in the document to extract topics. These methods do not consider word neighborhood and are based on exact word match. Graph-based extraction of topics is another popular approach (Ying et al., 2017 and Sayyadi & Raschid, 2013) for identifying the broad themes in documents. These methods are based on establishing a relationship between words/concepts using estimates such as co-occurrence, semantic similarity, etc. for extraction of prominent topics in a document. Variants of the above two approaches are popularly used for topic identification and available as off-the-shelf tools for identifying prominent words in the document.

We propose employing a variation of the graph-based method for identifying topics and utilizing it to obtain important segments of the judgment. Let $\mathfrak{L}=\left\{{\mathfrak{L}}_1,{\mathfrak{L}}_2\dots ,{\mathfrak{L}}_n\right\}$ be the set of ‘n’ legal judgments in the corpus. Let n(𝔏_i) be the set of sentences in the legal document 𝔏_i and let 𝔏_ij be the ‘j’th sentence of the ‘i’th legal judgment document. As the first step in the pre-processing of documents, we construct the base concept words as the nouns present in sentences in the judgment. Liu et al. (2018) propose extraction of keywords as basic concepts where authors demonstrate similarity estimation for news reports using concept interaction graph. Specific and distinctive characteristics of legal documents require a domain-specific approach for the extraction of concepts from the documents. While a person’s name may have a lot of relevance in a news report, it just represents a party (respondent or appellant) or a participant in the case and does not actually contribute to any legal concept present in the judgment. Therefore, we ignore references to specific people, place etc. which appear as proper nouns from the input in the document, and we define base word concept set of the ‘j’th sentence in the ‘i’th document, $\mathfrak{B}\left({\mathfrak{L}}_{ij}\right)$, as: (1)${\displaystyle \mathfrak{B}\left({\mathfrak{L}}_{ij}\right)=\left\{x\in {\mathfrak{L}}_{ij}\right|pos\left(x\right){=}^{{}^{\prime }}CommonNou{n}^{{}^{\prime }}andx\in \Im \left({\mathfrak{L}}_{ij}\right)}$

Here $pos\left(x\right)$stands for part of speech of the word x and ℑ(𝔏_ij) represents the important words in the sentence 𝔏_ij. We consider a common noun appearing in the sentences as concepts and construct a concept interaction graph using concept co-occurrences. However, we are selective about the nouns appearing in the concept graph and only allow important nouns to represent the document fragments. TF-IDF, term frequency-inverse document frequency method is the most fundamental weighing scheme used in an information retrieval system. TF-IDF computes weight for a term in a document collection by assessing its local relevance using term frequency within the document (TF) and global relevance by computing inverse document frequency for the entire document collection (Ramos, 2003). To assess this importance of the nouns we use TF-IDF model constructed for individual judgment by considering every sentence in a judgment as a separate document. The judgment, therefore, can be deemed to be a collection of documents (${\mathfrak{L}}_{ij},jϵ1\dots n\left({\mathfrak{L}}_i\right)$). Therefore $\Im \left({\mathfrak{L}}_{ij}\right)$ can be determined as: (2)${\displaystyle \Im \left({\mathfrak{L}}_{ij}\right)=\left\{x\in {\mathfrak{L}}_{ij}\right|tf\left(x,{\mathfrak{L}}_{ij}\right)\times idf\left(x,{\mathfrak{L}}_i\right)>\underset{\mathit{kϵ}\left[1,n\left({\mathfrak{L}}_i\right)\right]}{\mathit{mean}}\left(tf\left(x,{\mathfrak{L}}_{ik}\right)\times idf\left(x,{\mathfrak{L}}_i\right)\right)}$ where tf(x, 𝔏_ij) is the term frequency of the word ‘x’ in the sentence 𝔏_ij, $idf\left(x,{\mathfrak{L}}_i\right)$ measures the uniqueness of ‘x’ across the document 𝔏_i and the words having TF-IDF above the mean TF-IDF score over the document are considered important.

Identification of main concepts/topics of the document

Detection of related words in the judgment document is an important step and this is assessed based on the proximity of the base concept words. A concept graph 𝔊_i = (𝔙_i, 𝔈_i) of a legal document 𝔏_i,  is constructed using the base concept words s.t. ${\mathfrak{V}}_i={\bigcup }_{j\in \left[1,n\left({\mathfrak{L}}_i\right)\right]}\mathfrak{B}\left({\mathfrak{L}}_{ij}\right)$ and ${\mathfrak{E}}_i=\left\{\left(x,y\right)|co-occurrence\left(s,y\right)>3\right\}$. The set of vertices 𝔙_i is the set of all base concept words across all sentences in the document and two concept words nodes in the graph have an edge between them if their co-occurrence count is above 3 i.e., they appear together in at least three of the sentences. We use the count of co-occurrences as the strength of association between two concepts words. Less than 3 co-occurrences of concept words may represent mere coincidence and hence we do not deem such associations as strong enough for addition of edge in the graph. Figure 1 shows a concept graph constructed from a document fragment.

To discover important topics in the document we employ Louvain Modularity community detection algorithm (De Meo, 2011). The algorithm tries to locate communities by trying to maximize the modularity i.e., the ratio of the density of edges inside each community to the density of edges to nodes outside the community. The algorithm runs iteratively by first identifying small communities with high modularity and then proceeds to enlarge the communities by grouping them to achieve the maximum increase in modularity. We use the best partition method in the python networkx module, for detecting concepts in the document (Community detection for Networkx’s Documentation, 2010). Figure 2 shows an example of communities so evolved for a pair of judgments. Let m_i be the number of communities learnt for the document 𝔏_i and let the communities so detected be ℭ_i1, ℭ_i1, …, ℭ_imi. Each community thus identified is considered as a prominent concept which is represented by a set of words that formed the nodes in the initial concept graph.

Representative sentence selection and similarity estimation

Once the main concepts, represented as word communities, in the document are identified, for each concept the top five, most representative sentences for that concept are selected. TF-IDF scoring is used for this purpose. Each concept ℭ_ij, is a collection of words and can be considered as a document, similar to how each sentence in 𝔏_i is considered a document (Eq. (2)). Cosine similarity is computed between vectors representing each sentence in the judgment with the vector representing the concept. The five most similar sentences to the concept ℭ_ij are chosen as sentences representing that concept in the judgment 𝔏_i.

Our aim is to construct a vector representation of each concept occurring in a legal judgment which can capture the degree of occurrence of various ideas. These vector representations of concepts can ease the computation of similarity between two judgment documents. Let ${\mathfrak{S}}_{ij}^k$ be the kth representative sentence for the ‘j’th concept in ‘i’th legal document. The vector representation of each concept can be derived in various ways, the simplest one, being averaging the TF-IDF scores of ${\mathfrak{S}}_{ij}^k,k\in \left[1,5\right]$ i.e., averaging the TF-IDF scores for all sentences representative of a concept. However, we leverage on recent advances in neural networks, to take advantage of the potential knowledge captured by word embeddings. Word embeddings convert each word into a multi-dimensional vector of features in such a way that the vector representation of related words has high similarity. Word embeddings are often trained on huge real-world datasets and thus are able to capture the semantics very effectively. In addition, word embeddings obtained through popular methods like word2vec (Mikolov et al., 2013) have the property of additive compositionality i.e., the ability of the sum of word vectors of words composing a sentence/paragraph to preserve the semantics contained therein. Studies indicate that a word representation obtained using a combination of neural embeddings and TF-IDF provides is more effective (Lau & Baldwin, 2016) than the just the vector representations in many NLP tasks. Hence, we use IDF value for every word as weight applied to the vector of the word obtained using word2vec. We compute the vector 𝔚_ij corresponding to each concept ℭ_ij using two methods namely word2vec and IDF weighted word2vec and resultant vectors for these methods are computed using Eqs. (3) and (4) respectively. (3)${\displaystyle {\mathfrak{W}}_{ij}=\sum _{k=1}^5\sum _{x\in {\mathfrak{S}}_{ij}^k}word2vec\left(x\right)}$

and (4)${\displaystyle {\mathfrak{W}}_{ij}=\sum _{k=1}^5\sum _{x\in {\mathfrak{S}}_{ij}^k}word2vec\left(x\right)\ast IDF\left(x\right)}$

Here the summation involves vector addition of the word vectors of words belonging to each of the five representative sentences for the concept ℭ_ij.

The above-computed vector representation for each concept present in the judgment is finally used to compute the similarity between judgment documents. The notion of similarity among documents, sometimes, may not be sufficiently captured by single similarity value. Two documents may be similar to each other to different degrees when observed using different viewpoints. As an example, two legal documents may be similar because of commonalities in the case history but may be different in the way the cases were argued. On the other hand, two other legal documents may have nothing common in terms of the facts of the case but both may be overturning the judgment made by a lower court. To that extent, the two cases can be considered similar. When similarity computation is employed for judging the closeness of two documents, the context of the search may be unknown. In such cases estimating similarities using different notions and visualizing the same may be more helpful to the user than obtaining a single similarity score.

The ability to derive multiple vector representations for various concepts contained in a legal document in this proposed approach could aid in finding different levels of similarity between a pair of legal documents. Let 𝔏_a and 𝔏_b be two legal judgments consisting of 𝔫_a and 𝔫_b concepts respectively. We compute the similarity between each pair of concepts in 𝔏_a and 𝔏_b. Let sim(ℭ_ai, ℭ_bj) be the similarity between the ‘i’th and the ‘j’th concepts of the documents 𝔏_a and 𝔏_b, respectively. In this way, we obtain 𝔫_a × 𝔫_b similarity values. We use these similarity values to establish links between concepts across the two documents. For the proposed approach, we only allow each concept to participate in a maximum of one link. Modifications to this restriction are always possible and could possibly result in different similarity links and visualization result. The concepts in the two documents having the highest similarity value are linked using a similarity link. The documents which have already been linked are removed from further linking. This process is repeated taking the next two concepts one from each of the documents which are most similar and so on. The linking of highest matching concepts between a pair of judgments would be referred to as concept matches and an example of such a concept match is illustrated in Fig. 3. It is to be noted that in Figs. 2 and 3 only the concept words are shown (rather than the representative sentences) for ease of understanding. The strength of the lines connecting various concepts across the judgments are indicators of the level of match between the concepts. We present the following two examples to support the above explanation and to demonstrate how the proposed method is able to facilitate multi-level concept matching and visualization.

Example 1: A judgment pair discusses accident as a common theme but the facts in individual case result in multiple communities. Whereas there is a high similarity match in the discussion about the accident incident itself (Concept 1 in both judgments - shown as a bold link between the two), there is little match in Concept 2 of the pair, the first talks about charges of homicide whereas the second talks about negotiating the amount of compensation for the dependents of the deceased.

Example 2: A judgment pair with discussion on intellectual property rights (IPR) and copyright. The two cases present different dimensions of IPR. Case 1 discusses IPR with respect to copyright of literary work whereas case 2 discusses copyright on customer database as a business secret. Concept 2 and Concept 1 for the pair have a high likeness since these statements talk about property rights and infringement in general; the other two concepts in the judgments discuss copyright w.r.t. books while in the second judgment the unmatched concept discusses copyright on a database, trade, etc.

Example visualization for the above two situations is shown in Fig. 3A and Fig. 3B respectively. It is to be noted that the colors of the concept nodes are representative of the degree of closeness of the concepts.

The different levels of similarity so obtained can also be aggregated to compute a single similarity value which could be useful for finding all relevant documents to a given judgment. Given the various similarity values viewed from different perspectives between two judgments, we employ the Ordered Weight Averaging operator for aggregating the various similarity values into one. OWA is a family of aggregation operators introduced by Yager (2003) has a special application for multi-attribute decision-making problem especially in the presence of fuzzy data and allows for the incorporation of linguistic criteria for aggregation. Specifically, if there are items in a domain that need to be evaluated according to ‘p’ criteria $\left({\mathfrak{T}}_1,{\mathfrak{T}}_2,\dots ,{\mathfrak{T}}_p\right)$ s.t. 𝔗_j(item) is the extent to which ‘item’ satisfies the ‘j’th criterion, then it is possible to use the family of OWA aggregation operators to evaluate the degree to which ‘item’ satisfies “some criteria”, “all criteria”, “most criteria” etc. In the case of similarity estimation in the present case, we can consider the pair of judgments to be the item and the various possible criteria could be: degree of Match in facts of the case, degree of match in case citations, degree of Match in the defense counsel’s argument, etc. In this case, the pair of judgments would be evaluated according to each of the criteria and according to our choice of linguistic aggregation needed i.e., most, some, etc, the overall similarity can be computed. It is to be noted here that the set of criteria for legal judgments is not fixed and is determined for each document pair based on the concepts derived in each document.

The OWA operator (Yager, 2003) is defined as

Definition: OWA Operator A function f:Rⁿ → R is called on Ordered Weighted Averaging (OWA) operator of dimension ‘n’ if it has an associated weighting vector W of dimension ‘n’ such that:

${\displaystyle \left(1\right)\sum _{i=1}^n{W}_i=1}$ ${\displaystyle \left(2\right)W_i\in \left[0,1\right]\forall i=1,2\cdots ,n}$

Where, F is defined as $F\left(x_1,x_2,\dots ,x_n\right)={\sum }_{i=1}^n{W}_i{y}_i$, where y_i is the ‘i’th largest value in the set of elements {x₁, x₂, …, x_n}.

OWA can be used to emulate different aggregation operators such as max, min, average, etc, by adjusting the weights W_i, ∀i = 1, 2⋯, n, suitably. These linguist operators fall in between the extremes of “at least one” to “all”.

In the current work, we propose to use the “most” aggregation operator. In this paper, we just outline the method of arriving at the weights for the OWA operator and do not discuss the reasoning behind it. An in-depth presentation of the OWA operators is presented in (Carlsson & Fullér, 1996). If there are ‘p’ criteria for evaluating the similarity between a pair of documents (i.e., p concepts matches between a pair of documents), then we define an operator Q_most, corresponding to the linguistic quantifier “most” as $Q_{\text{most}}\left(\mathrm{x}\right)={\mathrm{x}}^2$. Then the weights for the OWA_most operator can be determined by the formula (Carlsson & Fullér, 1996) as: (5)${\displaystyle \mathrm{W}\left(\mathrm{i}\right)=\mathrm{Q}\left(\frac{\mathrm{i}}{\mathrm{p}}\right)-\mathrm{Q}\left(\frac{\mathrm{i}-1}{\mathrm{p}}\right)}$

Figure 4 depicts similarity estimation using OWA as described above for a sample pair of judgments. As shown in the figure, for the first judgment (Doc1), three concepts are identified which are represented using three corresponding set of words. For the second judgment (Doc2), two concepts are identified. The computation of similarity depicted in the figure is performed on the sentences representative of these concepts as explained above.

The similarity so computed for various documents can then be used to rank judgments in order of relevance to a query judgment.

Table 1 depicts the sample results obtained for a pair of judgments ranked as similar (ranked as 8 on a scale of 1–10) by human expert. Weight in the Table 1 represents the similarity of the sentence with the identified concept. Using the proposed approach of similarity estimation using OWA, a similarity score of 0.82 is obtained for this pair of judgments.

The following few sections present the efficacy of the proposed method using various experiments.

Experimental setup

Some of the judgments documents are extremely small and may not reveal any pattern. We considered 9,372 judgments with a length of more than 10 sentences for this work. These documents are cleaned by removing the metadata information about the date, judge’s names, bench details, etc. While this information may be required for searching a particular case, it doesn’t contribute to the similarity among case judgment. Judgments contain a lot of non-text information like section and rule numbers, specific number and naming conventions used for references that include special characters that need to be preserved. Such information poses challenges in the pre-processing task and demands a domain-specific pre-processing which is important for deciding similarity. Following pre-processing steps are used in our work

1. Preserve numbers and special characters wherever significant by removing space between word and number. Used for citation objects with numbers For Example section 23 converted to section23, clause 3(a) converted to clause 3a.

2. Use common nomenclature for citation objects(IPC <->Indian Penal Code, Constitution of India <->Indian Constitution etc.).( Guide to Legal Citation, 2010)

3. Perform generic linguistic pre-processing of case conversion, English stop word removal stemming and lemmatization, punctuation and number removal. Only numbers as words are removed i.e., section 23 retained but a number 456 removed.

4. Remove Legal stop words. Some words (e.g., petitioner, petition, respondent, court, etc.) appear in almost every judgment. We construct legal stop word set forming a list of words having the highest frequency across all documents and remove these words from the documents.

The set of 9,372 judgment documents pre-processed as above is used for training in the proposed work to obtain word embedding and TF-IDF weights for words which are used for calculation of similarity. We used Gensim package Word2Vec library (Gensim, 2014) for implementation. Word2Vec function is trained on pre-processed judgment corpus. The function results in a vector representation of every word of the considered documents. We experimented with different vector dimensions for training Word2Vec. Best results were obtained for vector dimension 100 which is used for all the experiments in this work. We used Gensim TF-IDF library (Gensim, 2014) for obtaining TF-IDF weights for the words in the document collection.