Edit on GitHub

Semantic Diff for SQL

by Iaroslav Zeigerman

Motivation

Software is constantly changing and evolving, and identifying what has changed and reviewing those changes is an integral part of the development process. SQL code is no exception to this.

Text-based diff tools such as git diff, when applied to a code base, have certain limitations. First, they can only detect insertions and deletions, not movements or updates of individual pieces of code. Second, such tools can only detect changes between lines of text, which is too coarse for something as granular and detailed as source code. Additionally, the outcome of such a diff is dependent on the underlying code formatting, and yields different results if the formatting should change.

Consider the following diff generated by Git:

Git diff output

Semantically the query hasn’t changed. The two arguments b and c have been swapped (moved), posing no impact on the output of the query. Yet Git replaced the whole affected expression alongside a bulk of unrelated elements.

The alternative to text-based diffing is to compare Abstract Syntax Trees (AST) instead. The main advantage of ASTs are that they are a direct product of code parsing, which represents the underlying code structure at any desired level of granularity. Comparing ASTs may yield extremely precise diffs; changes such as code movements and updates can also be detected. Even more importantly, this approach facilitates additional use cases beyond eyeballing two versions of source code side by side.

The use cases I had in mind for SQL when I decided to embark on this journey of semantic diffing were the following:

  • Query similarity score. Identifying which parts the two queries have in common to automatically suggest opportunities for consolidation, creation of intermediate/staging tables, and so on.
  • Differentiating between cosmetic / structural changes and functional ones. For example when a nested query is refactored into a common table expression (CTE), this kind of change doesn’t have any functional impact on either a query or its outcome.
  • Automatic suggestions about the need to retroactively backfill data. This is especially important for pipelines that populate very large tables for which restatement is a runtime-intensive procedure. The ability to discern between simple code movements and actual modifications can help assess the impact of a change and make suggestions accordingly.

The implementation discussed in this post is now a part of the SQLGlot library. You can find a complete source code in the diff.py module. The choice of SQLglot was an obvious one due to its simple but powerful API, lack of external dependencies and, more importantly, extensive list of supported SQL dialects.

The Search for a Solution

When it comes to any diffing tool (not just a semantic one), the primary challenge is to match as many elements of compared entities as possible. Once such a set of matching elements is available, deriving a sequence of changes becomes an easy task.

If our elements have unique identifiers associated with them (for example, an element’s ID in DOM), the matching problem is trivial. However, the SQL syntax trees that we are comparing have neither unique keys nor object identifiers that can be used for the purposes of matching. So, how do we suppose to find pairs of nodes that are related?

To better illustrate the problem, consider comparing the following SQL expressions: SELECT a + b + c, d, e and SELECT a - b + c, e, f. Matching individual nodes from respective syntax trees can be visualized as follows:

Figure 1: Example of node matching for two SQL expression trees Figure 1: Example of node matching for two SQL expression trees.

By looking at the figure of node matching for two SQL expression trees above, we conclude that the following changes should be captured by our solution:

  • Inserted nodes: Sub and f. These are the nodes from the target AST which do not have a matching node in the source AST.
  • Removed nodes: Add and d. These are the nodes from the source AST which do not have a counterpart in the target AST.
  • Remaining nodes must be identified as unchanged.

It should be clear at this point that if we manage to match nodes in the source tree with their counterparts in the target tree, then computing the diff becomes a trivial matter.

Naïve Brute-Force

The naïve solution would be to try all different permutations of node pair combinations, and see which set of pairs performs the best based on some type of heuristics. The runtime cost of such a solution quickly reaches the escape velocity; if both trees had only 10 nodes each, the number of such sets would approximately be 10! ^ 2 = 3.6M ^ 2 ~= 13 * 10^12. This is a very bad case of factorial complexity (to be precise, it’s actually much worse - O(n! ^ 2) - but I couldn’t come up with a name for it), so there is little need to explore this approach any further.

Myers Algorithm

After the naïve approach was proven to be infeasible, the next question I asked myself was “how does git diff work?”. This question led me to discover the Myers diff algorithm [1]. This algorithm has been designed to compare sequences of strings. At its core, it’s looking for the shortest path on a graph of possible edits that transform the first sequence into the second one, while heavily rewarding those paths that lead to longest subsequences of unchanged elements. There’s a lot of material out there describing this algorithm in greater detail. I found James Coglan’s series of blog posts to be the most comprehensive.

Therefore, I had this “brilliant” (actually not) idea to transform trees into sequences by traversing them in topological order, and then applying the Myers algorithm on resulting sequences while using a custom heuristics when checking the equality of two nodes. Unsurprisingly, comparing sequences of strings is quite different from comparing hierarchical tree structures, and by flattening trees into sequences, we lose a lot of relevant context. This resulted in a terrible performance of this algorithm on ASTs. It often matched completely unrelated nodes, even when the two trees were mostly the same, and produced extremely inaccurate lists of changes overall. After playing around with it a little and tweaking my equality heuristics to improve accuracy, I ultimately scrapped the whole implementation and went back to the drawing board.

Change Distiller

The algorithm I settled on at the end was Change Distiller, created by Fluri et al. [2], which in turn is an improvement over the core idea described by Chawathe et al. [3].

The algorithm consists of two high-level steps:

  1. Finding appropriate matchings between pairs of nodes that are part of compared ASTs. Identifying what is meant by “appropriate” matching is also a part of this step.
  2. Generating the so-called “edit script” from the matching set built in the 1st step. The edit script is a sequence of edit operations (for example, insert, remove, update, etc.) on individual tree nodes, such that when applied as transformations on the source AST, it eventually becomes the target AST. In general, the shorter the sequence, the better. The length of the edit script can be used to compare the performance of different algorithms, though this is not the only metric that matters.

The rest of this section is dedicated to the Python implementation of the steps above using the AST implementation provided by the SQLGlot library.

Building the Matching Set

Matching Leaves

We begin composing the matching set by matching the leaf nodes. Leaf nodes are the nodes that do not have any children nodes (such as literals, identifiers, etc.). In order to match them, we gather all the leaf nodes from the source tree and generate a cartesian product with all the leaves from the target tree, while comparing pairs created this way and assigning them a similarity score. During this stage, we also exclude pairs that don’t pass basic matching criteria. Then, we pick pairs that scored the highest while making sure that each node is matched no more than once.

Using the example provided at the beginning of the post, the process of building an initial set of candidate matchings can be seen on Figure 2.

Figure 2: Building a set of candidate matchings between leaf nodes. The third item in each triplet represents a similarity score between two nodes. Figure 2: Building a set of candidate matchings between leaf nodes. The third item in each triplet represents a similarity score between two nodes.

First, let’s analyze the similarity score. Then, we’ll discuss matching criteria.

The similarity score proposed by Fluri et al. [2] is a dice coefficient applied to bigrams of respective node values. A bigram is a sequence of two adjacent elements from a string computed in a sliding window fashion:

def bigram(string):
    count = max(0, len(string) - 1)
    return [string[i : i + 2] for i in range(count)]

For reasons that will become clear shortly, we actually need to compute bigram histograms rather than just sequences:

from collections import defaultdict

def bigram_histo(string):
    count = max(0, len(string) - 1)
    bigram_histo = defaultdict(int)
    for i in range(count):
        bigram_histo[string[i : i + 2]] += 1
    return bigram_histo

The dice coefficient formula looks like following:

Dice Coefficient

Where X is a bigram of the source node and Y is a bigram of the second one. What this essentially does is count the number of bigram elements the two nodes have in common, multiply it by 2, and then divide by the total number of elements in both bigrams. This is where bigram histograms come in handy:

def dice_coefficient(source, target):
    source_histo = bigram_histo(source.sql())
    target_histo = bigram_histo(target.sql())

    total_grams = (
        sum(source_histo.values()) + sum(target_histo.values())
    )
    if not total_grams:
        return 1.0 if source == target else 0.0

    overlap_len = 0
    overlapping_grams = set(source_histo) & set(target_histo)
    for g in overlapping_grams:
        overlap_len += min(source_histo[g], target_histo[g])

    return 2 * overlap_len / total_grams

To compute a bigram given a tree node, we first transform the node into its canonical SQL representation,so that the Literal(123) node becomes just “123” and the Identifier(“a”) node becomes just “a”. We also handle a scenario when strings are too short to derive bigrams. In this case, we fallback to checking the two nodes for equality.

Now when we know how to compute the similarity score, we can take care of the matching criteria for leaf nodes. In the original paper [2], the matching criteria is formalized as follows:

Matching criteria for leaf nodes

The two nodes are matched if two conditions are met:

  1. The node labels match (in our case labels are just node types).
  2. The similarity score for node values is greater than or equal to some threshold “f”. The authors of the paper recommend setting the value of “f” to 0.6.

With building blocks in place, we can now build a matching set for leaf nodes. First, we generate a list of candidates for matching:

from heapq import heappush, heappop

candidate_matchings = []
source_leaves = _get_leaves(self._source)
target_leaves = _get_leaves(self._target)
for source_leaf in source_leaves:
    for target_leaf in target_leaves:
        if _is_same_type(source_leaf, target_leaf):
            similarity_score = dice_coefficient(
                source_leaf, target_leaf
            )
            if similarity_score >= 0.6:
                heappush(
                    candidate_matchings,
                    (
                        -similarity_score,
                        len(candidate_matchings),
                        source_leaf,
                        target_leaf,
                    ),
                )

In the implementation above, we push each matching pair onto the heap to automatically maintain the correct order based on the assigned similarity score.

Finally, we build the initial matching set by picking leaf pairs with the highest score:

matching_set = set()
while candidate_matchings:
    _, _, source_leaf, target_leaf = heappop(candidate_matchings)
    if (
        source_leaf in unmatched_source_nodes
        and target_leaf in unmatched_target_nodes
    ):
        matching_set.add((source_leaf, target_leaf))
        unmatched_source_nodes.remove(source_leaf)
        unmatched_target_nodes.remove(target_leaf)

To finalize the matching set, we should now proceed with matching inner nodes.

Matching Inner Nodes

Matching inner nodes is quite similar to matching leaf nodes, with the following two distinctions:

  • Rather than ranking a set of possible candidates, we pick the first node pair that passes the matching criteria.
  • The matching criteria itself has been extended to account for the number of leaf nodes the pair of inner nodes have in common.

Figure 3: Matching inner nodes based on their type as well as how many of their leaf nodes have been previously matched. Figure 3: Matching inner nodes based on their type as well as how many of their leaf nodes have been previously matched.

Let’s start with the matching criteria. The criteria is formalized as follows:

Matching criteria for inner nodes

Alongside already familiar similarity score and node type criteria, there is a new one in the middle: the ratio of leaf nodes that the two nodes have in common must exceed some threshold “t”. The recommended value for “t” is also 0.6. Counting the number of common leaf nodes is pretty straightforward, since we already have the complete matching set for leaves. All we need to do is count how many matching pairs do leaf nodes from the two compared inner nodes form.

There are two additional heuristics associated with this matching criteria:

  • Inner node similarity weighting: if the similarity score between the node values doesn’t pass the threshold “f” but the ratio of common leaf nodes (“t”) is greater than or equal to 0.8, then the matching is considered successful.
  • The threshold “t” is reduced to 0.4 for inner nodes with the number of leaf nodes equal to 4 or less, in order to decrease the false negative rate for small subtrees.

We now only have to iterate through the remaining unmatched nodes and form matching pairs based on the outlined criteria:

leaves_matching_set = matching_set.copy()

for source_node in unmatched_source_nodes.copy():
    for target_node in unmatched_target_nodes:
        if _is_same_type(source_node, target_node):
            source_leaves = set(_get_leaves(source_node))
            target_leaves = set(_get_leaves(target_node))

            max_leaves_num = max(len(source_leaves), len(target_leaves))
            if max_leaves_num:
                common_leaves_num = sum(
                    1 if s in source_leaves and t in target_leaves else 0
                    for s, t in leaves_matching_set
                )
                leaf_similarity_score = common_leaves_num / max_leaves_num
            else:
                leaf_similarity_score = 0.0

            adjusted_t = (
                0.6
                if min(len(source_leaves), len(target_leaves)) > 4
                else 0.4
            )

            if leaf_similarity_score >= 0.8 or (
                leaf_similarity_score >= adjusted_t
                and dice_coefficient(source_node, target_node) >= 0.6
            ):
                matching_set.add((source_node, target_node))
                unmatched_source_nodes.remove(source_node)
                unmatched_target_nodes.remove(target_node)
                break

After the matching set is formed, we can proceed with generation of the edit script, which will be the algorithm’s output.

Generating the Edit Script

At this point, we should have the following 3 sets at our disposal:

  • The set of matched node pairs.
  • The set of remaining unmatched nodes from the source tree.
  • The set of remaining unmatched nodes from the target tree.

We can derive 3 kinds of edits from the matching set: either the node’s value was updated (Update), the node was moved to a different position within the tree (Move), or the node remained unchanged (Keep). Note that the Move case is not mutually exclusive with the other two. The node could have been updated or could have remained the same while at the same time its position within its parent node or the parent node itself could have changed. All unmatched nodes from the source tree are the ones that were removed (Remove), while unmatched nodes from the target tree are the ones that were inserted (Insert).

The latter two cases are pretty straightforward to implement:

edit_script = []

for removed_node in unmatched_source_nodes:
    edit_script.append(Remove(removed_node))
for inserted_node in unmatched_target_nodes:
    edit_script.append(Insert(inserted_node))

Traversing the matching set requires a little more thought:

for source_node, target_node in matching_set:
    if (
        not isinstance(source_node, LEAF_EXPRESSION_TYPES)
        or source_node == target_node
    ):
        move_edits = generate_move_edits(
            source_node, target_node, matching_set
        )
        edit_script.extend(move_edits)
        edit_script.append(Keep(source_node, target_node))
    else:
        edit_script.append(Update(source_node, target_node))

If a matching pair represents a pair of leaf nodes, we check if they are the same to decide whether an update took place. For inner node pairs, we also need to compare the positions of their respective children to detect node movements. Chawathe et al. [3] suggest applying the longest common subsequence (LCS) algorithm which, no surprise here, was described by Myers himself [1]. There is a small catch, however: instead of checking the equality of two children nodes, we need to check whether the two nodes form a pair that is a part of our matching set.

Now with this knowledge, the implementation becomes straightforward:

def generate_move_edits(source, target, matching_set):
    source_children = _get_child_nodes(source)
    target_children = _get_child_nodes(target)

    lcs = set(
        _longest_common_subsequence(
            source_children,
            target_children,
            lambda l, r: (l, r) in matching_set
        )
    )

    move_edits = []
    for node in source_children:
        if node not in lcs and node not in unmatched_source_nodes:
            move_edits.append(Move(node))

    return move_edits

I left out the implementation of the LCS algorithm itself here, but there are plenty of implementation choices out there that can be easily looked up.

Output

The implemented algorithm produces the output that resembles the following:

>>> from sqlglot import parse_one, diff
>>> diff(parse_one("SELECT a + b + c, d, e"), parse_one("SELECT a - b + c, e, f"))

Remove(Add)
Remove(Column(d))
Remove(Identifier(d))
Insert(Sub)
Insert(Column(f))
Insert(Identifier(f))
Keep(Select, Select)
Keep(Add, Add)
Keep(Column(a), Column(a))
Keep(Identifier(a), Identifier(a))
Keep(Column(b), Column(b))
Keep(Identifier(b), Identifier(b))
Keep(Column(c), Column(c))
Keep(Identifier(c), Identifier(c))
Keep(Column(e), Column(e))
Keep(Identifier(e), Identifier(e))

Note that the output above is abbreviated. The string representation of actual AST nodes is significantly more verbose.

The implementation works especially well when coupled with the SQLGlot’s query optimizer which can be used to produce canonical representations of compared queries:

>>> schema={"t": {"a": "INT", "b": "INT", "c": "INT", "d": "INT"}}
>>> source = """
... SELECT 1 + 1 + a
... FROM t
... WHERE b = 1 OR (c = 2 AND d = 3)
... """
>>> target = """
... SELECT 2 + a
... FROM t
... WHERE (b = 1 OR c = 2) AND (b = 1 OR d = 3)
... """
>>> optimized_source = optimize(parse_one(source), schema=schema)
>>> optimized_target = optimize(parse_one(target), schema=schema)
>>> edit_script = diff(optimized_source, optimized_target)
>>> sum(0 if isinstance(e, Keep) else 1 for e in edit_script)
0

Optimizations

The worst case runtime complexity of this algorithm is not exactly stellar: O(n^2 * log n^2). This is because of the leaf matching process, which involves ranking a cartesian product between all leaf nodes of compared trees. Unsurprisingly, the algorithm takes a considerable time to finish for bigger queries.

There are still a few basic things we can do in our implementation to help improve performance:

  • Refer to individual node objects using their identifiers (Python’s id()) instead of direct references in sets. This helps avoid costly recursive hash calculations and equality checks.
  • Cache bigram histograms to avoid computing them more than once for the same node.
  • Compute the canonical SQL string representation for each tree once while caching string representations of all inner nodes. This prevents redundant tree traversals when bigrams are computed.

At the time of writing only the first two optimizations have been implemented, so there is an opportunity to contribute for anyone who’s interested.

Alternative Solutions

This section is dedicated to solutions that I’ve investigated, but haven’t tried.

First, this section wouldn’t be complete without Tristan Hume’s blog post. Tristan’s solution has a lot in common with the Myers algorithm plus heuristics that is much more clever than what I came up with. The implementation relies on a combination of dynamic programming and A* search algorithm to explore the space of possible matchings and pick the best ones. It seemed to have worked well for Tistan’s specific use case, but after my negative experience with the Myers algorithm, I decided to try something different.

Another notable approach is the Gumtree algorithm by Falleri et al. [4]. I discovered this paper after I’d already implemented the algorithm that is the main focus of this post. In sections 5.2 and 5.3 of their paper, the authors compare the two algorithms side by side and claim that Gumtree is significantly better in terms of both runtime performance and accuracy when evaluated on 12 792 pairs of Java source files. This doesn’t surprise me, as the algorithm takes the height of subtrees into account. In my tests, I definitely saw scenarios in which this context would have helped. On top of that, the authors promise O(n^2) runtime complexity in the worst case which, given the Change Distiller's O(n^2 * log n^2), looks particularly tempting. I hope to try this algorithm out at some point, and there is a good chance you see me writing about it in my future posts.

Conclusion

The Change Distiller algorithm yielded quite satisfactory results in most of my tests. The scenarios in which it fell short mostly concerned identical (or very similar) subtrees located in different parts of the AST. In those cases, node mismatches were frequent and, as a result, edit scripts were somewhat suboptimal.

Additionally, the runtime performance of the algorithm leaves a lot to be desired. On trees with 1000 leaf nodes each, the algorithm takes a little under 2 seconds to complete. My implementation still has room for improvement, but this should give you a rough idea of what to expect. It appears that the Gumtree algorithm [4] can help address both of these points. I hope to find bandwidth to work on it soon and then compare the two algorithms side-by-side to find out which one performs better on SQL specifically. In the meantime, Change Distiller definitely gets the job done, and I can now proceed with applying it to some of the use cases I mentioned at the beginning of this post.

I’m also curious to learn whether other folks in the industry faced a similar problem, and how they approached it. If you did something similar, I’m interested to hear about your experience.

References

[1] Eugene W. Myers. An O(ND) Difference Algorithm and Its Variations. Algorithmica 1(2): 251-266 (1986)

[2] B. Fluri, M. Wursch, M. Pinzger, and H. Gall. Change Distilling: Tree differencing for fine-grained source code change extraction. IEEE Trans. Software Eng., 33(11):725–743, 2007.

[3] S.S. Chawathe, A. Rajaraman, H. Garcia-Molina, and J. Widom. Change Detection in Hierarchically Structured Information. Proc. ACM Sigmod Int’l Conf. Management of Data, pp. 493-504, June 1996

[4] Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, Martin Monperrus. Fine-grained and Accurate Source Code Differencing. Proceedings of the International Conference on Automated Software Engineering, 2014, Västeras, Sweden. pp.313-324, 10.1145/2642937.2642982. hal-01054552


  1"""
  2.. include:: ../posts/sql_diff.md
  3
  4----
  5"""
  6
  7from __future__ import annotations
  8
  9import typing as t
 10from collections import defaultdict
 11from dataclasses import dataclass
 12from heapq import heappop, heappush
 13
 14from sqlglot import Dialect, expressions as exp
 15from sqlglot.helper import ensure_list
 16
 17
 18@dataclass(frozen=True)
 19class Insert:
 20    """Indicates that a new node has been inserted"""
 21
 22    expression: exp.Expression
 23
 24
 25@dataclass(frozen=True)
 26class Remove:
 27    """Indicates that an existing node has been removed"""
 28
 29    expression: exp.Expression
 30
 31
 32@dataclass(frozen=True)
 33class Move:
 34    """Indicates that an existing node's position within the tree has changed"""
 35
 36    expression: exp.Expression
 37
 38
 39@dataclass(frozen=True)
 40class Update:
 41    """Indicates that an existing node has been updated"""
 42
 43    source: exp.Expression
 44    target: exp.Expression
 45
 46
 47@dataclass(frozen=True)
 48class Keep:
 49    """Indicates that an existing node hasn't been changed"""
 50
 51    source: exp.Expression
 52    target: exp.Expression
 53
 54
 55if t.TYPE_CHECKING:
 56    from sqlglot._typing import T
 57
 58    Edit = t.Union[Insert, Remove, Move, Update, Keep]
 59
 60
 61def diff(
 62    source: exp.Expression,
 63    target: exp.Expression,
 64    matchings: t.List[t.Tuple[exp.Expression, exp.Expression]] | None = None,
 65    delta_only: bool = False,
 66    **kwargs: t.Any,
 67) -> t.List[Edit]:
 68    """
 69    Returns the list of changes between the source and the target expressions.
 70
 71    Examples:
 72        >>> diff(parse_one("a + b"), parse_one("a + c"))
 73        [
 74            Remove(expression=(COLUMN this: (IDENTIFIER this: b, quoted: False))),
 75            Insert(expression=(COLUMN this: (IDENTIFIER this: c, quoted: False))),
 76            Keep(
 77                source=(ADD this: ...),
 78                target=(ADD this: ...)
 79            ),
 80            Keep(
 81                source=(COLUMN this: (IDENTIFIER this: a, quoted: False)),
 82                target=(COLUMN this: (IDENTIFIER this: a, quoted: False))
 83            ),
 84        ]
 85
 86    Args:
 87        source: the source expression.
 88        target: the target expression against which the diff should be calculated.
 89        matchings: the list of pre-matched node pairs which is used to help the algorithm's
 90            heuristics produce better results for subtrees that are known by a caller to be matching.
 91            Note: expression references in this list must refer to the same node objects that are
 92            referenced in source / target trees.
 93        delta_only: excludes all `Keep` nodes from the diff.
 94        kwargs: additional arguments to pass to the ChangeDistiller instance.
 95
 96    Returns:
 97        the list of Insert, Remove, Move, Update and Keep objects for each node in the source and the
 98        target expression trees. This list represents a sequence of steps needed to transform the source
 99        expression tree into the target one.
100    """
101    matchings = matchings or []
102    matching_ids = {id(n) for pair in matchings for n in pair}
103
104    def compute_node_mappings(
105        original: exp.Expression, copy: exp.Expression
106    ) -> t.Dict[int, exp.Expression]:
107        return {
108            id(old_node): new_node
109            for old_node, new_node in zip(original.walk(), copy.walk())
110            if id(old_node) in matching_ids
111        }
112
113    source_copy = source.copy()
114    target_copy = target.copy()
115
116    node_mappings = {
117        **compute_node_mappings(source, source_copy),
118        **compute_node_mappings(target, target_copy),
119    }
120    matchings_copy = [(node_mappings[id(s)], node_mappings[id(t)]) for s, t in matchings]
121
122    return ChangeDistiller(**kwargs).diff(
123        source_copy,
124        target_copy,
125        matchings=matchings_copy,
126        delta_only=delta_only,
127    )
128
129
130# The expression types for which Update edits are allowed.
131UPDATABLE_EXPRESSION_TYPES = (
132    exp.Alias,
133    exp.Boolean,
134    exp.Column,
135    exp.DataType,
136    exp.Lambda,
137    exp.Literal,
138    exp.Table,
139    exp.Window,
140)
141
142IGNORED_LEAF_EXPRESSION_TYPES = (exp.Identifier,)
143
144
145class ChangeDistiller:
146    """
147    The implementation of the Change Distiller algorithm described by Beat Fluri and Martin Pinzger in
148    their paper https://ieeexplore.ieee.org/document/4339230, which in turn is based on the algorithm by
149    Chawathe et al. described in http://ilpubs.stanford.edu:8090/115/1/1995-46.pdf.
150    """
151
152    def __init__(self, f: float = 0.6, t: float = 0.6) -> None:
153        self.f = f
154        self.t = t
155        self._sql_generator = Dialect().generator()
156
157    def diff(
158        self,
159        source: exp.Expression,
160        target: exp.Expression,
161        matchings: t.List[t.Tuple[exp.Expression, exp.Expression]] | None = None,
162        delta_only: bool = False,
163    ) -> t.List[Edit]:
164        matchings = matchings or []
165        pre_matched_nodes = {id(s): id(t) for s, t in matchings}
166        if len({n for pair in pre_matched_nodes.items() for n in pair}) != 2 * len(matchings):
167            raise ValueError("Each node can be referenced at most once in the list of matchings")
168
169        self._source = source
170        self._target = target
171        self._source_index = {
172            id(n): n for n in self._source.bfs() if not isinstance(n, IGNORED_LEAF_EXPRESSION_TYPES)
173        }
174        self._target_index = {
175            id(n): n for n in self._target.bfs() if not isinstance(n, IGNORED_LEAF_EXPRESSION_TYPES)
176        }
177        self._unmatched_source_nodes = set(self._source_index) - set(pre_matched_nodes)
178        self._unmatched_target_nodes = set(self._target_index) - set(pre_matched_nodes.values())
179        self._bigram_histo_cache: t.Dict[int, t.DefaultDict[str, int]] = {}
180
181        matching_set = self._compute_matching_set() | {(s, t) for s, t in pre_matched_nodes.items()}
182        return self._generate_edit_script(matching_set, delta_only)
183
184    def _generate_edit_script(
185        self,
186        matching_set: t.Set[t.Tuple[int, int]],
187        delta_only: bool,
188    ) -> t.List[Edit]:
189        edit_script: t.List[Edit] = []
190        for removed_node_id in self._unmatched_source_nodes:
191            edit_script.append(Remove(self._source_index[removed_node_id]))
192        for inserted_node_id in self._unmatched_target_nodes:
193            edit_script.append(Insert(self._target_index[inserted_node_id]))
194        for kept_source_node_id, kept_target_node_id in matching_set:
195            source_node = self._source_index[kept_source_node_id]
196            target_node = self._target_index[kept_target_node_id]
197            if (
198                not isinstance(source_node, UPDATABLE_EXPRESSION_TYPES)
199                or source_node == target_node
200            ):
201                edit_script.extend(
202                    self._generate_move_edits(source_node, target_node, matching_set)
203                )
204                if not delta_only:
205                    edit_script.append(Keep(source_node, target_node))
206            else:
207                edit_script.append(Update(source_node, target_node))
208
209        return edit_script
210
211    def _generate_move_edits(
212        self, source: exp.Expression, target: exp.Expression, matching_set: t.Set[t.Tuple[int, int]]
213    ) -> t.List[Move]:
214        source_args = [id(e) for e in _expression_only_args(source)]
215        target_args = [id(e) for e in _expression_only_args(target)]
216
217        args_lcs = set(_lcs(source_args, target_args, lambda l, r: (l, r) in matching_set))
218
219        move_edits = []
220        for a in source_args:
221            if a not in args_lcs and a not in self._unmatched_source_nodes:
222                move_edits.append(Move(self._source_index[a]))
223
224        return move_edits
225
226    def _compute_matching_set(self) -> t.Set[t.Tuple[int, int]]:
227        leaves_matching_set = self._compute_leaf_matching_set()
228        matching_set = leaves_matching_set.copy()
229
230        ordered_unmatched_source_nodes = {
231            id(n): None for n in self._source.bfs() if id(n) in self._unmatched_source_nodes
232        }
233        ordered_unmatched_target_nodes = {
234            id(n): None for n in self._target.bfs() if id(n) in self._unmatched_target_nodes
235        }
236
237        for source_node_id in ordered_unmatched_source_nodes:
238            for target_node_id in ordered_unmatched_target_nodes:
239                source_node = self._source_index[source_node_id]
240                target_node = self._target_index[target_node_id]
241                if _is_same_type(source_node, target_node):
242                    source_leaf_ids = {id(l) for l in _get_leaves(source_node)}
243                    target_leaf_ids = {id(l) for l in _get_leaves(target_node)}
244
245                    max_leaves_num = max(len(source_leaf_ids), len(target_leaf_ids))
246                    if max_leaves_num:
247                        common_leaves_num = sum(
248                            1 if s in source_leaf_ids and t in target_leaf_ids else 0
249                            for s, t in leaves_matching_set
250                        )
251                        leaf_similarity_score = common_leaves_num / max_leaves_num
252                    else:
253                        leaf_similarity_score = 0.0
254
255                    adjusted_t = (
256                        self.t if min(len(source_leaf_ids), len(target_leaf_ids)) > 4 else 0.4
257                    )
258
259                    if leaf_similarity_score >= 0.8 or (
260                        leaf_similarity_score >= adjusted_t
261                        and self._dice_coefficient(source_node, target_node) >= self.f
262                    ):
263                        matching_set.add((source_node_id, target_node_id))
264                        self._unmatched_source_nodes.remove(source_node_id)
265                        self._unmatched_target_nodes.remove(target_node_id)
266                        ordered_unmatched_target_nodes.pop(target_node_id, None)
267                        break
268
269        return matching_set
270
271    def _compute_leaf_matching_set(self) -> t.Set[t.Tuple[int, int]]:
272        candidate_matchings: t.List[t.Tuple[float, int, int, exp.Expression, exp.Expression]] = []
273        source_leaves = list(_get_leaves(self._source))
274        target_leaves = list(_get_leaves(self._target))
275        for source_leaf in source_leaves:
276            for target_leaf in target_leaves:
277                if _is_same_type(source_leaf, target_leaf):
278                    similarity_score = self._dice_coefficient(source_leaf, target_leaf)
279                    if similarity_score >= self.f:
280                        heappush(
281                            candidate_matchings,
282                            (
283                                -similarity_score,
284                                -_parent_similarity_score(source_leaf, target_leaf),
285                                len(candidate_matchings),
286                                source_leaf,
287                                target_leaf,
288                            ),
289                        )
290
291        # Pick best matchings based on the highest score
292        matching_set = set()
293        while candidate_matchings:
294            _, _, _, source_leaf, target_leaf = heappop(candidate_matchings)
295            if (
296                id(source_leaf) in self._unmatched_source_nodes
297                and id(target_leaf) in self._unmatched_target_nodes
298            ):
299                matching_set.add((id(source_leaf), id(target_leaf)))
300                self._unmatched_source_nodes.remove(id(source_leaf))
301                self._unmatched_target_nodes.remove(id(target_leaf))
302
303        return matching_set
304
305    def _dice_coefficient(self, source: exp.Expression, target: exp.Expression) -> float:
306        source_histo = self._bigram_histo(source)
307        target_histo = self._bigram_histo(target)
308
309        total_grams = sum(source_histo.values()) + sum(target_histo.values())
310        if not total_grams:
311            return 1.0 if source == target else 0.0
312
313        overlap_len = 0
314        overlapping_grams = set(source_histo) & set(target_histo)
315        for g in overlapping_grams:
316            overlap_len += min(source_histo[g], target_histo[g])
317
318        return 2 * overlap_len / total_grams
319
320    def _bigram_histo(self, expression: exp.Expression) -> t.DefaultDict[str, int]:
321        if id(expression) in self._bigram_histo_cache:
322            return self._bigram_histo_cache[id(expression)]
323
324        expression_str = self._sql_generator.generate(expression)
325        count = max(0, len(expression_str) - 1)
326        bigram_histo: t.DefaultDict[str, int] = defaultdict(int)
327        for i in range(count):
328            bigram_histo[expression_str[i : i + 2]] += 1
329
330        self._bigram_histo_cache[id(expression)] = bigram_histo
331        return bigram_histo
332
333
334def _get_leaves(expression: exp.Expression) -> t.Iterator[exp.Expression]:
335    has_child_exprs = False
336
337    for node in expression.iter_expressions():
338        if not isinstance(node, IGNORED_LEAF_EXPRESSION_TYPES):
339            has_child_exprs = True
340            yield from _get_leaves(node)
341
342    if not has_child_exprs:
343        yield expression
344
345
346def _is_same_type(source: exp.Expression, target: exp.Expression) -> bool:
347    if type(source) is type(target):
348        if isinstance(source, exp.Join):
349            return source.args.get("side") == target.args.get("side")
350
351        if isinstance(source, exp.Anonymous):
352            return source.this == target.this
353
354        return True
355
356    return False
357
358
359def _parent_similarity_score(
360    source: t.Optional[exp.Expression], target: t.Optional[exp.Expression]
361) -> int:
362    if source is None or target is None or type(source) is not type(target):
363        return 0
364
365    return 1 + _parent_similarity_score(source.parent, target.parent)
366
367
368def _expression_only_args(expression: exp.Expression) -> t.List[exp.Expression]:
369    args: t.List[t.Union[exp.Expression, t.List]] = []
370    if expression:
371        for a in expression.args.values():
372            args.extend(ensure_list(a))
373    return [
374        a
375        for a in args
376        if isinstance(a, exp.Expression) and not isinstance(a, IGNORED_LEAF_EXPRESSION_TYPES)
377    ]
378
379
380def _lcs(
381    seq_a: t.Sequence[T], seq_b: t.Sequence[T], equal: t.Callable[[T, T], bool]
382) -> t.Sequence[t.Optional[T]]:
383    """Calculates the longest common subsequence"""
384
385    len_a = len(seq_a)
386    len_b = len(seq_b)
387    lcs_result = [[None] * (len_b + 1) for i in range(len_a + 1)]
388
389    for i in range(len_a + 1):
390        for j in range(len_b + 1):
391            if i == 0 or j == 0:
392                lcs_result[i][j] = []  # type: ignore
393            elif equal(seq_a[i - 1], seq_b[j - 1]):
394                lcs_result[i][j] = lcs_result[i - 1][j - 1] + [seq_a[i - 1]]  # type: ignore
395            else:
396                lcs_result[i][j] = (
397                    lcs_result[i - 1][j]
398                    if len(lcs_result[i - 1][j]) > len(lcs_result[i][j - 1])  # type: ignore
399                    else lcs_result[i][j - 1]
400                )
401
402    return lcs_result[len_a][len_b]  # type: ignore
@dataclass(frozen=True)
class Insert:
19@dataclass(frozen=True)
20class Insert:
21    """Indicates that a new node has been inserted"""
22
23    expression: exp.Expression

Indicates that a new node has been inserted

Insert(expression: sqlglot.expressions.Expression)
@dataclass(frozen=True)
class Remove:
26@dataclass(frozen=True)
27class Remove:
28    """Indicates that an existing node has been removed"""
29
30    expression: exp.Expression

Indicates that an existing node has been removed

Remove(expression: sqlglot.expressions.Expression)
@dataclass(frozen=True)
class Move:
33@dataclass(frozen=True)
34class Move:
35    """Indicates that an existing node's position within the tree has changed"""
36
37    expression: exp.Expression

Indicates that an existing node's position within the tree has changed

Move(expression: sqlglot.expressions.Expression)
@dataclass(frozen=True)
class Update:
40@dataclass(frozen=True)
41class Update:
42    """Indicates that an existing node has been updated"""
43
44    source: exp.Expression
45    target: exp.Expression

Indicates that an existing node has been updated

@dataclass(frozen=True)
class Keep:
48@dataclass(frozen=True)
49class Keep:
50    """Indicates that an existing node hasn't been changed"""
51
52    source: exp.Expression
53    target: exp.Expression

Indicates that an existing node hasn't been changed

def diff( source: sqlglot.expressions.Expression, target: sqlglot.expressions.Expression, matchings: Optional[List[Tuple[sqlglot.expressions.Expression, sqlglot.expressions.Expression]]] = None, delta_only: bool = False, **kwargs: Any) -> List[Union[Insert, Remove, Move, Update, Keep]]:
 62def diff(
 63    source: exp.Expression,
 64    target: exp.Expression,
 65    matchings: t.List[t.Tuple[exp.Expression, exp.Expression]] | None = None,
 66    delta_only: bool = False,
 67    **kwargs: t.Any,
 68) -> t.List[Edit]:
 69    """
 70    Returns the list of changes between the source and the target expressions.
 71
 72    Examples:
 73        >>> diff(parse_one("a + b"), parse_one("a + c"))
 74        [
 75            Remove(expression=(COLUMN this: (IDENTIFIER this: b, quoted: False))),
 76            Insert(expression=(COLUMN this: (IDENTIFIER this: c, quoted: False))),
 77            Keep(
 78                source=(ADD this: ...),
 79                target=(ADD this: ...)
 80            ),
 81            Keep(
 82                source=(COLUMN this: (IDENTIFIER this: a, quoted: False)),
 83                target=(COLUMN this: (IDENTIFIER this: a, quoted: False))
 84            ),
 85        ]
 86
 87    Args:
 88        source: the source expression.
 89        target: the target expression against which the diff should be calculated.
 90        matchings: the list of pre-matched node pairs which is used to help the algorithm's
 91            heuristics produce better results for subtrees that are known by a caller to be matching.
 92            Note: expression references in this list must refer to the same node objects that are
 93            referenced in source / target trees.
 94        delta_only: excludes all `Keep` nodes from the diff.
 95        kwargs: additional arguments to pass to the ChangeDistiller instance.
 96
 97    Returns:
 98        the list of Insert, Remove, Move, Update and Keep objects for each node in the source and the
 99        target expression trees. This list represents a sequence of steps needed to transform the source
100        expression tree into the target one.
101    """
102    matchings = matchings or []
103    matching_ids = {id(n) for pair in matchings for n in pair}
104
105    def compute_node_mappings(
106        original: exp.Expression, copy: exp.Expression
107    ) -> t.Dict[int, exp.Expression]:
108        return {
109            id(old_node): new_node
110            for old_node, new_node in zip(original.walk(), copy.walk())
111            if id(old_node) in matching_ids
112        }
113
114    source_copy = source.copy()
115    target_copy = target.copy()
116
117    node_mappings = {
118        **compute_node_mappings(source, source_copy),
119        **compute_node_mappings(target, target_copy),
120    }
121    matchings_copy = [(node_mappings[id(s)], node_mappings[id(t)]) for s, t in matchings]
122
123    return ChangeDistiller(**kwargs).diff(
124        source_copy,
125        target_copy,
126        matchings=matchings_copy,
127        delta_only=delta_only,
128    )

Returns the list of changes between the source and the target expressions.

Examples:
>>> diff(parse_one("a + b"), parse_one("a + c"))
[
    Remove(expression=(COLUMN this: (IDENTIFIER this: b, quoted: False))),
    Insert(expression=(COLUMN this: (IDENTIFIER this: c, quoted: False))),
    Keep(
        source=(ADD this: ...),
        target=(ADD this: ...)
    ),
    Keep(
        source=(COLUMN this: (IDENTIFIER this: a, quoted: False)),
        target=(COLUMN this: (IDENTIFIER this: a, quoted: False))
    ),
]
Arguments:
  • source: the source expression.
  • target: the target expression against which the diff should be calculated.
  • matchings: the list of pre-matched node pairs which is used to help the algorithm's heuristics produce better results for subtrees that are known by a caller to be matching. Note: expression references in this list must refer to the same node objects that are referenced in source / target trees.
  • delta_only: excludes all Keep nodes from the diff.
  • kwargs: additional arguments to pass to the ChangeDistiller instance.
Returns:

the list of Insert, Remove, Move, Update and Keep objects for each node in the source and the target expression trees. This list represents a sequence of steps needed to transform the source expression tree into the target one.

IGNORED_LEAF_EXPRESSION_TYPES = (<class 'sqlglot.expressions.Identifier'>,)
class ChangeDistiller:
146class ChangeDistiller:
147    """
148    The implementation of the Change Distiller algorithm described by Beat Fluri and Martin Pinzger in
149    their paper https://ieeexplore.ieee.org/document/4339230, which in turn is based on the algorithm by
150    Chawathe et al. described in http://ilpubs.stanford.edu:8090/115/1/1995-46.pdf.
151    """
152
153    def __init__(self, f: float = 0.6, t: float = 0.6) -> None:
154        self.f = f
155        self.t = t
156        self._sql_generator = Dialect().generator()
157
158    def diff(
159        self,
160        source: exp.Expression,
161        target: exp.Expression,
162        matchings: t.List[t.Tuple[exp.Expression, exp.Expression]] | None = None,
163        delta_only: bool = False,
164    ) -> t.List[Edit]:
165        matchings = matchings or []
166        pre_matched_nodes = {id(s): id(t) for s, t in matchings}
167        if len({n for pair in pre_matched_nodes.items() for n in pair}) != 2 * len(matchings):
168            raise ValueError("Each node can be referenced at most once in the list of matchings")
169
170        self._source = source
171        self._target = target
172        self._source_index = {
173            id(n): n for n in self._source.bfs() if not isinstance(n, IGNORED_LEAF_EXPRESSION_TYPES)
174        }
175        self._target_index = {
176            id(n): n for n in self._target.bfs() if not isinstance(n, IGNORED_LEAF_EXPRESSION_TYPES)
177        }
178        self._unmatched_source_nodes = set(self._source_index) - set(pre_matched_nodes)
179        self._unmatched_target_nodes = set(self._target_index) - set(pre_matched_nodes.values())
180        self._bigram_histo_cache: t.Dict[int, t.DefaultDict[str, int]] = {}
181
182        matching_set = self._compute_matching_set() | {(s, t) for s, t in pre_matched_nodes.items()}
183        return self._generate_edit_script(matching_set, delta_only)
184
185    def _generate_edit_script(
186        self,
187        matching_set: t.Set[t.Tuple[int, int]],
188        delta_only: bool,
189    ) -> t.List[Edit]:
190        edit_script: t.List[Edit] = []
191        for removed_node_id in self._unmatched_source_nodes:
192            edit_script.append(Remove(self._source_index[removed_node_id]))
193        for inserted_node_id in self._unmatched_target_nodes:
194            edit_script.append(Insert(self._target_index[inserted_node_id]))
195        for kept_source_node_id, kept_target_node_id in matching_set:
196            source_node = self._source_index[kept_source_node_id]
197            target_node = self._target_index[kept_target_node_id]
198            if (
199                not isinstance(source_node, UPDATABLE_EXPRESSION_TYPES)
200                or source_node == target_node
201            ):
202                edit_script.extend(
203                    self._generate_move_edits(source_node, target_node, matching_set)
204                )
205                if not delta_only:
206                    edit_script.append(Keep(source_node, target_node))
207            else:
208                edit_script.append(Update(source_node, target_node))
209
210        return edit_script
211
212    def _generate_move_edits(
213        self, source: exp.Expression, target: exp.Expression, matching_set: t.Set[t.Tuple[int, int]]
214    ) -> t.List[Move]:
215        source_args = [id(e) for e in _expression_only_args(source)]
216        target_args = [id(e) for e in _expression_only_args(target)]
217
218        args_lcs = set(_lcs(source_args, target_args, lambda l, r: (l, r) in matching_set))
219
220        move_edits = []
221        for a in source_args:
222            if a not in args_lcs and a not in self._unmatched_source_nodes:
223                move_edits.append(Move(self._source_index[a]))
224
225        return move_edits
226
227    def _compute_matching_set(self) -> t.Set[t.Tuple[int, int]]:
228        leaves_matching_set = self._compute_leaf_matching_set()
229        matching_set = leaves_matching_set.copy()
230
231        ordered_unmatched_source_nodes = {
232            id(n): None for n in self._source.bfs() if id(n) in self._unmatched_source_nodes
233        }
234        ordered_unmatched_target_nodes = {
235            id(n): None for n in self._target.bfs() if id(n) in self._unmatched_target_nodes
236        }
237
238        for source_node_id in ordered_unmatched_source_nodes:
239            for target_node_id in ordered_unmatched_target_nodes:
240                source_node = self._source_index[source_node_id]
241                target_node = self._target_index[target_node_id]
242                if _is_same_type(source_node, target_node):
243                    source_leaf_ids = {id(l) for l in _get_leaves(source_node)}
244                    target_leaf_ids = {id(l) for l in _get_leaves(target_node)}
245
246                    max_leaves_num = max(len(source_leaf_ids), len(target_leaf_ids))
247                    if max_leaves_num:
248                        common_leaves_num = sum(
249                            1 if s in source_leaf_ids and t in target_leaf_ids else 0
250                            for s, t in leaves_matching_set
251                        )
252                        leaf_similarity_score = common_leaves_num / max_leaves_num
253                    else:
254                        leaf_similarity_score = 0.0
255
256                    adjusted_t = (
257                        self.t if min(len(source_leaf_ids), len(target_leaf_ids)) > 4 else 0.4
258                    )
259
260                    if leaf_similarity_score >= 0.8 or (
261                        leaf_similarity_score >= adjusted_t
262                        and self._dice_coefficient(source_node, target_node) >= self.f
263                    ):
264                        matching_set.add((source_node_id, target_node_id))
265                        self._unmatched_source_nodes.remove(source_node_id)
266                        self._unmatched_target_nodes.remove(target_node_id)
267                        ordered_unmatched_target_nodes.pop(target_node_id, None)
268                        break
269
270        return matching_set
271
272    def _compute_leaf_matching_set(self) -> t.Set[t.Tuple[int, int]]:
273        candidate_matchings: t.List[t.Tuple[float, int, int, exp.Expression, exp.Expression]] = []
274        source_leaves = list(_get_leaves(self._source))
275        target_leaves = list(_get_leaves(self._target))
276        for source_leaf in source_leaves:
277            for target_leaf in target_leaves:
278                if _is_same_type(source_leaf, target_leaf):
279                    similarity_score = self._dice_coefficient(source_leaf, target_leaf)
280                    if similarity_score >= self.f:
281                        heappush(
282                            candidate_matchings,
283                            (
284                                -similarity_score,
285                                -_parent_similarity_score(source_leaf, target_leaf),
286                                len(candidate_matchings),
287                                source_leaf,
288                                target_leaf,
289                            ),
290                        )
291
292        # Pick best matchings based on the highest score
293        matching_set = set()
294        while candidate_matchings:
295            _, _, _, source_leaf, target_leaf = heappop(candidate_matchings)
296            if (
297                id(source_leaf) in self._unmatched_source_nodes
298                and id(target_leaf) in self._unmatched_target_nodes
299            ):
300                matching_set.add((id(source_leaf), id(target_leaf)))
301                self._unmatched_source_nodes.remove(id(source_leaf))
302                self._unmatched_target_nodes.remove(id(target_leaf))
303
304        return matching_set
305
306    def _dice_coefficient(self, source: exp.Expression, target: exp.Expression) -> float:
307        source_histo = self._bigram_histo(source)
308        target_histo = self._bigram_histo(target)
309
310        total_grams = sum(source_histo.values()) + sum(target_histo.values())
311        if not total_grams:
312            return 1.0 if source == target else 0.0
313
314        overlap_len = 0
315        overlapping_grams = set(source_histo) & set(target_histo)
316        for g in overlapping_grams:
317            overlap_len += min(source_histo[g], target_histo[g])
318
319        return 2 * overlap_len / total_grams
320
321    def _bigram_histo(self, expression: exp.Expression) -> t.DefaultDict[str, int]:
322        if id(expression) in self._bigram_histo_cache:
323            return self._bigram_histo_cache[id(expression)]
324
325        expression_str = self._sql_generator.generate(expression)
326        count = max(0, len(expression_str) - 1)
327        bigram_histo: t.DefaultDict[str, int] = defaultdict(int)
328        for i in range(count):
329            bigram_histo[expression_str[i : i + 2]] += 1
330
331        self._bigram_histo_cache[id(expression)] = bigram_histo
332        return bigram_histo

The implementation of the Change Distiller algorithm described by Beat Fluri and Martin Pinzger in their paper https://ieeexplore.ieee.org/document/4339230, which in turn is based on the algorithm by Chawathe et al. described in http://ilpubs.stanford.edu:8090/115/1/1995-46.pdf.

ChangeDistiller(f: float = 0.6, t: float = 0.6)
153    def __init__(self, f: float = 0.6, t: float = 0.6) -> None:
154        self.f = f
155        self.t = t
156        self._sql_generator = Dialect().generator()
f
t
def diff( self, source: sqlglot.expressions.Expression, target: sqlglot.expressions.Expression, matchings: Optional[List[Tuple[sqlglot.expressions.Expression, sqlglot.expressions.Expression]]] = None, delta_only: bool = False) -> List[Union[Insert, Remove, Move, Update, Keep]]:
158    def diff(
159        self,
160        source: exp.Expression,
161        target: exp.Expression,
162        matchings: t.List[t.Tuple[exp.Expression, exp.Expression]] | None = None,
163        delta_only: bool = False,
164    ) -> t.List[Edit]:
165        matchings = matchings or []
166        pre_matched_nodes = {id(s): id(t) for s, t in matchings}
167        if len({n for pair in pre_matched_nodes.items() for n in pair}) != 2 * len(matchings):
168            raise ValueError("Each node can be referenced at most once in the list of matchings")
169
170        self._source = source
171        self._target = target
172        self._source_index = {
173            id(n): n for n in self._source.bfs() if not isinstance(n, IGNORED_LEAF_EXPRESSION_TYPES)
174        }
175        self._target_index = {
176            id(n): n for n in self._target.bfs() if not isinstance(n, IGNORED_LEAF_EXPRESSION_TYPES)
177        }
178        self._unmatched_source_nodes = set(self._source_index) - set(pre_matched_nodes)
179        self._unmatched_target_nodes = set(self._target_index) - set(pre_matched_nodes.values())
180        self._bigram_histo_cache: t.Dict[int, t.DefaultDict[str, int]] = {}
181
182        matching_set = self._compute_matching_set() | {(s, t) for s, t in pre_matched_nodes.items()}
183        return self._generate_edit_script(matching_set, delta_only)