Ephemeral data handling in microservices with Tquery

The match operator

The purpose of the match operator is to select trees in an array a according to a criterion φ, which can be (from left to right): (i) the Boolean truth, (ii) the existence of a path p in t, (iii) the equality between the application of a path p on t and a given array a, (iv) the equality between the applications of two paths p₁ and p₂ on t, and the logic connectives (v) negation, (vi) conjunction, and (vii) disjunction.

Example Here and in the following sections, we draw our examples from Listing 2. There, we see the match operator used twice: the first at Line 3 and the second at Line 11. Here, we focus on the example at Line 3. We comment the execution of Line 11 in ‘The group operator’, since we use it to filter out the unnecessary values from the — sl— data structure before the application of the group.

At Line 3, we use a match to filter — tmp— from those trees that do not correspond to the time range of interest. For convenience, we report Line 3 of Listing 2 in the snippet below.

 
 
 
  ____________________________________________________________________________________________________________ 
 
  3 tmp ← μ ( tmp , ( date  =20201128 ∨ date  =20201129 ∨ date  =20201130)  )    

The execution takes as input the data structure — tmp— presented in Listing 1 and assigns to it the resulting data structure:

 
 
 
  ______________________________________________________________________________________________________________________________________ 
 
  [ υ { date : [ 20201128 { } ] , t : [ 36 { } ] , hr : [ 66 { } ] } , 
  υ { date : [ 20201129 { } ] , t : [ 36 { } ] , hr : [ 65 { } ] } , 
   υ { date : [ 20201130 { } ] , t : [ 37 { } ] , hr : [ 67 { } ] } ]    

Semantics When applied to an array a, the match operator returns a new array in the shape of a but including only its elements that satisfy φ. If no element matches the criterion (and also in the case that a = α), the operator returns an empty array [ ] . ${\displaystyle \mu \left(\alpha ,\phi \right)=\left[\right]\mu \left(\left[t\right]::a,\phi \right)=\left\{\begin{array}{c}\left[t\right]::\mu \left(a,\phi \right)\text{if}t\vDash \phi \hfill \\ \mu \left(a,\phi \right)\text{if}\mathtt{\#}a>0\hfill \\ \left[\right]\text{otherwise}\hfill \end{array}\right.}$

The semantics of t⊧φ is defined by the Boolean expressions below. ${\displaystyle t\vDash \phi \text{holds iff}\left\{\begin{array}{c}\phi =true\hfill \\ \phi =\left(\exists p\right)\wedge {\left[\left[p\right]\right]}^t\ne \alpha \hfill \\ \phi =\left(p=a\right)\wedge {\left[\left[p\right]\right]}^t=a\hfill \\ \phi =\left(p_1=p_2\right)\wedge {\left[\left[p_1\right]\right]}^t={\left[\left[p_2\right]\right]}^t\hfill \\ \phi =\left(\neg {\phi }^{\prime }\right)\wedge t⁄\vDash {\phi }^{\prime }\hfill \\ \phi =\left({\phi }_1\wedge {\phi }_2\right)\wedge \left(t\vDash {\phi }_1\wedge t\vDash {\phi }_2\right)\hfill \\ \phi =\left({\phi }_1\vee {\phi }_2\right)\wedge \left(t\vDash {\phi }_1\vee t\vDash {\phi }_2\right)\hfill \end{array}\right.}$

Example: semantics At Line 3 of Listing 2, the match evaluates all trees inside — tmp— and verifies which one of the sub-conditions hold for each element of — tmp—. In the case of — tmp—[1], the criterion is not satisfied and thus the value is discarded. Next, — tmp—[2] satisfies the first criterion

 
 date = 20201128)    

 
 
 
 
   \(\  date } =\  20201129 } \) 

 
 
 
 
   \( date } =\  20201130 } \) 

The unwind operator

The purpose of the unwind operator is to unfold the elements of an array a under a given path p.

Example We exemplify the usage of unwind reporting Line 9 of Listing 2 in the snippet below and later showing the result of its application.

 
 
 
  _______________________________________________________________________ 
 
  9 sl ← ω ( sl , M.D.L  )    

The unwind operator takes as input the sleep logs — sl— (as retrieved from the invocation of the — getSleepPatterns— operation at Line 8, and represented at Lines 7–16 of Listing 1). In the snippet, we update the content of — sl— to contain the new data structure, shown below.

 
 
 
  ________________________________________________________________________________________________________________________ 
 
  [ υ { y : [ 2020 { } ] , M : [ υ { m : [ 11 { } ] , D : [ υ { d : [ 27 { } ] , 
   L : [ υ { s : [ ' 23:33 ' { } ] , e : [ ' 07:04 ' { } ] , q : [ ' poor ' { } ] 
   } ] } ] } ] } , 
   υ { y : [ 2020 { } ] , M : [ υ { m : [ 11 { } ] , D : [ υ { d : [ 28 { } ] , 
   L : [ υ { s : [ ' 21:13 ' { } ] , e : [ ' 09:34 ' { } ] , q : [ ' good ' { } ] 
   } ] } ] } ] } , 
   υ { y : [ 2020 { } ] , M : [ υ { m : [ 11 { } ] , D : [ υ { d : [ 29 { } ] , 
   L : [ υ { s : [ ' 21:01 ' { } ] , e : [ ' 03:12 ' { } ] , q : [ ' good ' { } ] 
   } ] } ] } ] } , 
   ... ]    

Semantics To define the semantics of the unwind operator ω, we introduce an auxiliary operator, called unwind expansion operator and we indicate it with ueo(t, a, k) (read “unwind t on a under k”). Informally, ueo(t, a, k) returns an array of trees with cardinality #a where each element has the shape of t except that label k points to the corresponding index-wise element in a.

Formally, given a tree t, an array a, and a key k: ${\displaystyle \mathsf{ueo}\left(t,a,k\right)=\left\{\begin{array}{cc}\text{[}b\left(\left(\text{{}{k}_i:a_i{\text{}}}_i\setminus \text{{}k:k\left(t\right)\text{}}\right)\cup \text{{}k:\text{[}{t}^{\prime }\text{]}\text{}}\right)\text{]}::\mathsf{ueo}\left(t,a^{\prime },k\right)\hfill & \text{if}a=\text{[}{t}^{\prime }\text{]}::a^{\prime }\hfill \\ \hfill & \wedge t=b\text{{}{k}_i:a_i{\text{}}}_i\hfill \\ \text{[}\text{]}\hfill & \text{otherwise}\hfill \end{array}\right.}$ Then, the formal definition of ω(a, p) is ${\displaystyle \omega \left(a,p\right)=\left\{\begin{array}{cc}\mathsf{ueo}\left(t,\omega \left({\left[\left[\underset{¯}{k.\varepsilon }\right]\right]}^t,p^{\prime }\right),k\right)::\omega \left(a^{\prime },p\right)\hfill & \text{if}p=e.p^{\prime }\wedge e\downarrow k\wedge a=\text{[}t\text{]}::a^{\prime }\hfill \\ a\hfill & \text{if}p=\varepsilon \hfill \\ \text{[}\text{]}\hfill & \text{otherwise}\hfill \end{array}\right.}$

Essentially, the semantics of the unwind operator follows two inductive directions: one on arrays and the other on paths. Hence, to simplify the explanation of the semantics, we describe it following a spatial interpretation of the two directions: the induction on arrays is the “breadth” of the expansion while the induction on paths represents its “depth”.

The first part of the breadth expansion corresponds to the induction over the array a, which results in the concatenation of the inductive application of the depth expansion of p over each element t of a. In turn, the depth expansion consists of a nested depth expansion with a breadth one. The depth expansion is represented by $\omega \left({\left[\left[\underset{¯}{k.\varepsilon }\right]\right]}^t,p^{\prime }\right)$, which corresponds to the application of the unwind operator with path p′—the suffix of k in p—and on the array of subtrees found in t under the current path fragment k. The breadth expansion (which complements the breadth expansion on the array a) uses the unwind expansion operator (ueo) to apply the result of the nested depth expansion on all elements found under k in t.

Example: semantics We now report excerpts of the execution of the unwind operator at Line 9 of Listing 2 to exemplify both the unfolding of the breadth and depth expansions.

We remind that — sl— has the shape reported in Line 7 in Listing 1 and that the application at Line 9 of Listing 2 “unwinds” the — sl— data structure with path $\underset{¯}{M.D.L}$.

The first expansion we perform is the breadth expansion over the array — sl—. Since — sl— just contains one tree, i.e., that for sleep logs of 2020, we just have one application of the ueo operator (the empty array  [] at the right of the concatenation operator :: results from the “otherwise” branch of the definition of the unwind and from— sl— being structurally equivalent to sl[1]:: []).

 
 
 
   _______________________________________________________________________________________ 
 
   ueo (sl [1], ω([[ M.ɛ  ]]sl [1] , D.L  ), M) :: [ ] 
    

Then, we show the “depth” part of the expansion, by focusing on the terminal part of the application of the ueo operator. Specifically, we concentrate on the tree corresponding to the sleep logs of day 2020-11-29, found at Line 11 of Listing 1 and aliased with the tree t₂₉. Formally, the expansion corresponds to the application $\mathsf{ueo}\left(t_{29},{\left[\left[\underset{¯}{L.\varepsilon }\right]\right]}^{t_{29}},L\right)$ of the terminal node L in path $\underset{¯}{M.D.L}$.

 
 
 
  _______________________________________________________________________________________________________________________________________ 
 
  ueo ( t29 , [[ L.ɛ]] t29 , L ) ⇒ 
   [ ( ν { d : [ 29 { } ] , L : [ ... ] } \ ν { L : [ ... ] } ) ∪ ν { L : [ ν { s : [ ' 21:01 ' { } ] , e : [ ' 03:12 ' 
    { } ] , q : [ ' good ' { } ] } ] } ] :: 
    [ ( ν { d : [ 29 { } ] , L : [ ... ] } \ ν { L : [ ... ] } ) ∪ ν { L : [ ν { s : [ ' 03:36 ' { } ] , e : [ ' 09:58 ' 
    { } ] , q : [ ' good ' { } ] } ] } ]    

Above, for each element of the array pointed by L, we create a new structure where we replace the original array associated with the key L with a new array containing only one element. For instance, the first element of the result takes the original structure found under D ( [ν  {d:[29{}], L: [... ] } ]) and updates it to contain only the element

 
 
   υ { s : [ ' 21:01 ' { } ] , e : [ ' 03:12 ' { } ] , q : [ ' good ' { } ] } 

The project operator

The purpose of the project operator is to modify the trees in an array a by projecting nodes, renaming node labels, or introducing new nodes, as described in the sequence of elements Π, which are either a path p or an injection (〈〉) of a value definition d into a path.

A value definition d can be (in the grammar, from left to right): (i) a value, (ii) a path, (iii) an array of value definitions, (iv) a criterion (φ) (cf. ‘The match operator”) or (v) a ternary expression on a criterion and two value definitions.

Example As done for the other operators, we draw our examples from Listing 2, where we have four usages of the project operator, the first at Line 5, the second at Line 10, the third at Line 13, and the fourth at Line 15. Here, we focus on the second example, at Line 10, reported in the snippet below. We comment on the others when exemplifying the lookup operator in ‘The lookup operator’.

 
 
 
  _________________________________________________________________________________________________________________________ 
 
  10 sl ← π ( sl , ( y  〈〉 year  , M.m  〈〉 month  , M.D.d  〈〉 day  , M.D.L.q  〈〉 quality 
) ) 
 
    

The projection at Line 10 takes the — sl— data structure resulting from the application of the unwind at Line 9 and performs a sequence of renaming over all tress within — sl—. For each tree, we perform the rename of the node y in year by moving the content of path $\underset{¯}{y}$ into the node corresponding to path $\underset{¯}{year}$, represented by the fragment $\underset{¯}{y}〈〉\underset{¯}{year}$. Similarly, we move the content of $\underset{¯}{M.m}$ under $\underset{¯}{month}$, of $\underset{¯}{M.D.d}$ under $\underset{¯}{day}$, and of $\underset{¯}{M.D.L.q}$ under $\underset{¯}{quality}$. The result of the projection is the following flattened structure:

 
 
 
  _________________________________________________________________________________________________________________________ 
 
  [ υ { year : [ 2020 { } ] , month : [ 11 { } ] , day : [ 27 { } ] , quality : [ ' 
 
   poor ' { } ] } , 
    υ { year : [ 2020 { } ] , month : [ 11 { } ] , day : [ 28 { } ] , quality : [ ' 
    good ' { } ] } , 
    υ { year : [ 2020 { } ] , month : [ 11 { } ] , day : [ 29 { } ] , quality : [ ' 
    good ' { } ] } , 
    ... ] 
    

Semantics We start by defining the auxiliary operators we use in the definition of the project. Auxiliary operators π(a, p) and π(t, p) formalise the application of a branch selection over a path p respectively over an array and a tree. Then, we define the auxiliary operator eval(d, t), which returns the array resulting from the evaluation of a value definition d over a tree t. Finally, we report the projection of an injection of a value definition d into a path p over a tree t, i.e., π(t, d〈〉p).

The projection π(a, p) for a path p over an array a results in an array whose elements are the projection for p of the elements of a: ${\displaystyle \pi \left(a,p\right)=\pi \left(\text{[}{t}_1,\dots ,t_n\text{]},p\right)=\text{[}\pi \left(t_1,p\right),\dots ,\pi \left(t_n,p\right)\text{]}}$

The projection π(t, p) for a path p over a tree t implements the actual semantics of branch selection, where, given a path e.p′ with e↓k, we remove all the branches  but  and continue to apply the projection for the continuation p′ over the (array of) sub-trees under k in t (i.e., ${\left[\left[\underset{¯}{k.\varepsilon }\right]\right]}^t$). Formally: ${\displaystyle \pi \left(t,p\right)=\left\{\begin{array}{cc}\upsilon \text{{}k:\pi \left(\left[\left[\underset{¯}{k.ϵ}\right]\right]t,p^{\prime }\right)\text{}}\hfill & \text{if}{\left[\left[p\right]\right]}^t\ne \alpha \wedge p=e.p^{\prime }\wedge e\downarrow k\hfill \\ t\hfill & \text{if}p=\varepsilon \hfill \\ \tau \hfill & \text{otherwise}\hfill \end{array}\right.}$

The operator eval(d, t) evaluates the value definition d over the tree t and returns an array containing the result of the evaluation. Formally: ${\displaystyle \mathsf{eval}\left(d,t\right)=\left\{\begin{array}{cc}\text{[}d\text{{}\text{}}\text{]}\hfill & \text{if}d\in V\hfill \\ {\left[\left[d\right]\right]}^t\hfill & \text{if}d\in P\hfill \\ \mathsf{eval}\left(d,t\right)::\mathsf{eval}\left(d^{\prime },t\right)\hfill & \text{if}d=\text{[}d\text{]}::d^{\prime }\hfill \\ \text{[}t\vDash \phi \text{{}\text{}}\text{]}\hfill & \text{if}d=\phi \hfill \\ \mathsf{eval}\left(d_1,t\right)\hfill & \text{if}d=\phi ?d_1:d_2\wedge t\vDash \phi \hfill \\ \mathsf{eval}\left(d_2,t\right)\hfill & \text{if}d=\phi ?d_1:d_2\wedge t⁄\vDash \phi \hfill \\ \alpha \hfill & \text{otherwise}\hfill \end{array}\right.}$

The projection π(t, d〈〉p) of the injection of the evaluation of a value definition d on a tree t into a path p results in a new tree where we find the evaluation of d on t under p. ${\displaystyle \pi \left(t,d〈〉p\right)=\left\{\begin{array}{cc}\upsilon \text{{}k:\text{[}\pi \left(t,d〈〉p^{\prime }\right)\text{]}\text{}}\hfill & \text{if}p=e.p^{\prime }\wedge e\downarrow k\wedge \mathsf{eval}\left(d,t\right)\ne \alpha \hfill \\ \upsilon \text{{}k:\text{eval}\left(d,t\right)\text{}}\hfill & \text{if}p=e.\varepsilon \wedge e\downarrow k\wedge \mathsf{eval}\left(d,t\right)\ne \alpha \hfill \\ \tau \hfill & \text{otherwise}\hfill \end{array}\right.}$

Before formalising the projection, we report the auxiliary operator ⊕ to merge arrays and trees—we use the operator to merge the result of sequences of projections in the definition of π(t, Π). ${\displaystyle \begin{array}{c}\hfill \left(\text{[}t\text{]}::a\right)\oplus \left(\text{[}{t}^{\prime }\text{]}::a^{\prime }\right)=\text{[}t\oplus t^{\prime }\text{]}::a\oplus a^{\prime }t\oplus \tau =ta\oplus \text{[}\text{]}=\text{[}\text{]}\oplus a=a\oplus \alpha =\alpha \oplus a=a\hfill \end{array}}$ ${\displaystyle \frac{t=b\text{{}{k}_i:a_i{\text{}}}_{i\in I}\wedge t^{\prime }=b\text{{}{k}_j:a_j{\text{}}}_{j\in J}}{t\oplus t^{\prime }=b\text{{}{k}_h:k_h\left(t\right)\oplus k_h\left(t^{\prime }\right){\text{}}}_{h\in I\cup J}}\frac{b\ne b^{\prime }}{b\text{{}{k}_i:a_i{\text{}}}_i\oplus b^{\prime }\text{{}{k}_j:a_j{\text{}}}_j=\tau }}$

To conclude, we first report the application of the projection to a tree t, π(t, Π), which merges the results of projections in Π over t into a single tree. Second, we report the application of the projection to an array a, π(a, Π), which corresponds to the application of the projection to all elements of a. Respectively, we formally write: ${\displaystyle \pi \left(t,\Pi \right)=\left\{\begin{array}{cc}\pi \left(t,p\right)\oplus \left(\pi \left(t,{\Pi }^{\prime }\right)\right)\hfill & \text{if}\Pi =p,{\Pi }^{\prime }\hfill \\ \pi \left(t,d〈〉p\right)\oplus \left(\pi \left(t,{\Pi }^{\prime }\right)\right)\hfill & \text{if}\Pi =d〈〉p,{\Pi }^{\prime }\hfill \\ \pi \left(t,p\right)\hfill & \text{if}\Pi =p\hfill \\ \pi \left(t,d〈〉p\right)\hfill & \text{if}\Pi =d〈〉p\hfill \end{array}\right.}$ and ${\displaystyle \pi \left(a,\Pi \right)=\pi \left(\text{[}{t}_1,\dots ,t_n\text{]},\Pi \right)=\text{[}\pi \left(t_1,\Pi \right),\dots ,\pi \left(t_n,\Pi \right)\text{]}\pi \left(\text{[}\text{]},\Pi \right)=\pi \left(\alpha ,\Pi \right)=\text{[}\text{]}}$

Example: semantics We report the execution of the project at Line 10 of Listing 2. We take — sl— as returned after the application of the unwind operator described in ‘The unwind operator’. For brevity, we represent the — sl— data structure as the concatenation of its elements, i.e., sl = sl[1]::sl[2]::⋯.

 
 
 
   _______________________________________________________________________________________________________________________ 
 
   π ( sl [1] :: sl [2] :: ⋅⋅⋅ , 
 
   (y 〈〉 year  , M.m  〈〉 month  , M.D.d  〈〉 day  , M.D.L.q  〈〉 quality  )) ⇒ 
 
  [ π( sl [1],(y 〈〉 year  , M.m  〈〉 month  , M.D.d  〈〉 day  , M.D.L.q  〈〉 quality  )) , 
 
   π( sl [2],(y 〈〉 year  , M.m  〈〉 month  , M.D.d  〈〉 day  , M.D.L.q  〈〉 quality  )) , ... ]    

We continue showing the projection of the first element in a, sl[1] (the projection on the other elements follows the same structure)

 
 
 
   _______________________________________________________________________________________________________________________ 
 
   π( sl [1],(y 〈〉 year  , M.m  〈〉 month  , M.D.d  〈〉 day  , M.D.L.q  〈〉 quality  )) ⇒ 
 π( sl [1],y 〈〉 year  ) ⊕ π( sl [1],M.m 〈〉 month  ) ⊕ π( sl [1],M.D.d 〈〉 day  ) ⊕ π( sl 
   [1],M.D.L.q 〈〉 quality  ) 

Finally, we show the unfolding of the first two projections from the left, above, i.e., those for $\underset{¯}{y}〈〉\underset{¯}{year}$ and for $\underset{¯}{M.m}〈〉\underset{¯}{month}$, and their merge ⊕ (the remaining ones unfold similarly).

 
 
 
    π( sl [1],y 〈〉 year  ) ⊕ π( sl [1],M.m 〈〉 month  ) 
     ⇒ υ { year : π( sl [1],y ) } ⊕ υ { month : π( sl [1],M.m ) } 
   ⇒ υ { year : eval (y , sl [1]) } ⊕ υ { month : eval (M.m , sl [1]) } 
   ⇒ υ { year : [[ y ]]sl [1] } ⊕ υ { month : [[ M.m ]]sl [1] } 
   ⇒ υ { year : [ 2020 { } ] } ⊕ υ { month : [ 11 { } ] } 
   ⇒ υ { year : [ 2020 { } ] , month : [ 11 { } ] }    

The group operator

The purpose of the group operator is to group the trees in an array a according to a specification Γ′ and to aggregate the values of the grouped trees according to the specification Γ. Both Γ and Γ′, respectively the aggregation and the grouping set, are sequences of elements of the form p〈〉p′ where p is a path in the input trees, and p′ a path in the output trees.

Note that Γ includes both fragments of the shape p and p〈〉p′. Here, the former is syntactic sugar for the latter, where both paths are the same. Therefore, we assume to apply the semantics of the group operator only with the de-sugared form γ(a, Γ, Γ′) = γ(a, exp(Γ), exp(Γ′)), where ${\displaystyle \mathsf{exp}\left({\Gamma }_1,{\Gamma }_2\right)=\mathsf{exp}\left({\Gamma }_1\right),\mathsf{exp}\left({\Gamma }_2\right)\mathsf{exp}\left(p\right)=p〈〉p\mathsf{exp}\left(q〈〉p\right)=q〈〉p}$

Example Drawing from Listing 2, we have two applications of the group operator, one at Line 4 and the second at Line 12. Since the two applications are similar, we just focus on the latter (reported below), leaving the comment on the second to ‘The lookup operator’.

 
 
 
12 sl ← γ ( sl , ( quality  ) , () )    

As stated above, the aggregation set expands from $\underset{¯}{quality}$ to the de-sugared form $\underset{¯}{quality}〈〉\underset{¯}{quality}$.

The group operator applies on the data structure in — sl— which, at Line 11, we filtered with the match operator to only contain values corresponding to the dates 2020-11-29 and 2020-11-30. The new data structure, copied into — sl— and reported below, is essentially the aggregation under the node quality of the filtered sleep recordings.

 
 
 
  _______________________________________________________________________________________________________________________ 
 
  [ υ { quality : [ ' good ' { } , ' good ' { } , ' poor ' { } , ' good ' { } ] } 
  ]    

To make for a more comprehensive illustration, in this section we consider an alternative version of the example above, where we want to use the group operator to group the values by $\underset{¯}{day}$, $\underset{¯}{month}$, and $\underset{¯}{year}$ and aggregate the values of the sleep $\underset{¯}{quality}$. Concretely, we do this by updating the command found at Line 12 with the sequence of paths replacing the third parameter, which in the original we left empty.

 
 
 
  _________________________________________________________________________________________ 
 
  sl ← γ ( sl , ( quality  ) , ( day  , month  , year  ) )    

As stated, the paths $\underset{¯}{quality}$, $\underset{¯}{day}$, $\underset{¯}{month}$, and $\underset{¯}{year}$ respectively expand to $\underset{¯}{quality}〈〉\underset{¯}{quality}$, $\underset{¯}{day}〈〉\underset{¯}{day}$, $\underset{¯}{month}〈〉\underset{¯}{month}$, and $\underset{¯}{year}〈〉\underset{¯}{year}$.

The main detail we want to notice here is that, by grouping the values by $\underset{¯}{year}$, $\underset{¯}{month}$, and $\underset{¯}{day}$, we only aggregate logs relative to the same day.

 
 
 
  ______________________________________________________________________________________________________________________ 
 
  [ υ { year : [ 2020 { } ] , month : [ 11 { } ] , day : [ 29 { } ] , quality : [  ' good  ' { } ,  ' 
     good  ' { } ] } , 
    υ { year : [ 2020 { } ] , month : [ 11 { } ] , day : [ 30 { } ] , quality : [  ' poor  ' { } ,  ' 
     good  ' { } ] } 
   ]    

Semantics We start by reminding the shape of the de-sugared syntax of the group operator. ${\displaystyle \gamma \left(a,\Gamma ,{\Gamma }^{\prime }\right)=\gamma \left(a,\mathsf{exp}\left(\Gamma \right),\mathsf{exp}\left({\Gamma }^{\prime }\right)\right)=\gamma \left(a,{q_1〈p_1,\dots ,q_n〉p_n}_{\text{aggregation set}\mathcal{A}},{s_1〈r_1,\dots ,s_m〉r_m}_{\text{grouping set}\mathcal{G}}\right)}$

Intuitively, the group operator performs the following actions:

1. it groups together those trees in a that (1) have the maximal number of existing paths from the grouping set s₁, …, s_m and (2) whose values under those paths coincide;

2. it projects the values in the grouped trees from s₁, …, s_m to the corresponding paths r₁, …, r_m;

3. it aggregates all the values in the grouped trees found under the paths q₁, …, q_n from the aggregation set;

4. it projects the aggregated values from q₁, …, q_n into the corresponding paths p₁, …, p_n.

Formally, let S = {s₁, …, s_m} be the set of left elements in the injections of the sequence in the grouping set and let Σ be the power-set 2^S of paths in S so that ${\displaystyle \Sigma =\left\{\varnothing ,\left\{s_1\right\},\left\{s_2\right\},\left\{s_3\right\},\dots ,\left\{s_1,s_2\right\},\left\{s_1,s_3\right\},\dots ,\left\{s_1,\dots ,s_m\right\}\right\}=\left\{{\sigma }_1,\dots ,{\sigma }_k\right\}}$

We define the auxiliary operator exists which takes S and an element σ ∈ Σ and builds the existence-match-query formula of the paths in S w.r.t. the combination identified by σ. ${\displaystyle \mathsf{exists}\left(\sigma ,S\right)=\left\{\begin{array}{cc}\mathtt{true}\hfill & \text{if}S=\varnothing \hfill \\ \exists s\wedge \mathsf{exists}\left(\sigma ,S\setminus \left\{s\right\}\right)\hfill & \text{let}s\in S\text{and}s\in \sigma \hfill \\ \neg \exists s\wedge \mathsf{exists}\left(\sigma ,S\setminus \left\{s\right\}\right)\hfill & \text{let}s\in S\text{and}s⁄\in \sigma \hfill \end{array}\right.}$

We use the exists operator to perform part 1) of Item (a), i.e., grouping those trees in a so that the trees in the same group have the same set of existing and non-existing paths from s₁, …, s_m. The part operator (presented below) performs part 2) of Item (a), which is the partition of the trees grouped by the exists operator so that the values in their existing paths in s₁, …, s_m coincide.

We now define the semantics of the group operator and then present the semantics of the part operator. In the remainder, to make the definitions more intuitive, we alias the aggregation set with $\mathcal{A}$ and the grouping set with $\mathcal{G}$. Let, k = |Σ|, we write ${\displaystyle \gamma \left(a,\mathcal{A},\mathcal{G}\right)=\mathsf{part}\left(\mu \left(a,\mathsf{exists}\left({\sigma }_1,S\right)\right),{\sigma }_1,\mathcal{A},\mathcal{G}\right)::\cdots ::\mathsf{part}\left(\mu \left(a,\mathsf{exists}\left({\sigma }_k,S\right)\right),{\sigma }_k,\mathcal{A},\mathcal{G}\right)}$

As mentioned, the part operator finds the elements of a which should be grouped together according to $\mathcal{G}$ (among those selected through σ). In the definition, we delegate the actual grouping to the other auxiliary operator group, which (as hinted in Item (b)) projects the partitioned values from S into the corresponding destination path r₁, r₂, … in $\mathcal{G}$. The group operator also performs the aggregation of the values found in q₁, q₂, … (Item (c)) and it projects them under the corresponding destination path p₁, p₂, … (Item (d)).

In the semantics of the part operator, we assume to extend the set difference ∖ to arrays, so that a∖a′ returns a copy of a without the elements found in a′ (preserving their relative order). We also assume to have a variant of the match operator μ^id(a, φ) that, instead of returning the array of trees in a that match the criterion φ, it returns the array of their indexes in a. ${\displaystyle \mathsf{part}\left(a,\sigma ,\mathcal{A},\mathcal{G}\right)=\left\{\begin{array}{cc}a\hfill & \text{if}a=\text{[}\text{]}\hfill \\ \mathsf{group}\left(a,\sigma ,\mathcal{A},\mathcal{G}\right)\hfill & \text{if}\sigma =0̸\hfill \\ \hfill & \text{otherwise, let}\sigma =\left\{s_1,\dots ,s_i\right\},\hfill \\ \mathsf{group}\left(\text{[}a\left[j\right],\dots ,a\left[k\right]\text{]},\sigma ,\mathcal{A},\mathcal{G}\right)::\mathsf{part}\left(\text{[}a\left[f\right],\dots ,a\left[g\right]\text{]},\sigma ,\mathcal{A},\mathcal{G}\right)\hfill & {\mu }^{id}\left(a,{\bigwedge }_{j=1}^i{s}_j={\left[\left[s_j\right]\right]}^{a\left[1\right]}\right)=\text{[}j,\dots ,k\text{]},\hfill \\ \hfill & \text{[}f,\dots ,g\text{]}=\text{[}1,\dots ,\text{\#}a\text{]}\setminus \text{[}j,\dots ,k\text{]}\hfill \end{array}\right.}$

Finally, we report below the definition of the group operator. There, the last case is where we aggregate the values found in the array a following the paths in $\mathcal{A}$, and we combine them with the grouped values from $\mathcal{G}$ by using the project operator. The aggregation of the values in a is done by invoking the group operator on the second case. The second case applies when σ = ∅ (i.e., when no path S is selected for grouping). The result of the application of the second case is an array containing one tree that combines the values of the array a following the paths in $\mathcal{A}$. To aggregate the values, we use the auxiliary tree variant of the project operator (π(t, Π), cf. ‘The project operator’) to project each value for a given path q into its corresponding path p in $\mathcal{A}$. ${\displaystyle \mathsf{group}\left(a,\sigma ,\mathcal{A},\mathcal{G}\right)=\left\{\begin{array}{c}a\text{if}a=\text{[}\text{]}\hfill \\ \text{[}\pi \left(\tau ,\underset{aggregation}{\underbrace{{\eta }_1〈p_1,\dots ,{\eta }_n〉p_n}}\right)\text{]}\text{if}\sigma =0̸,\text{let}\mathcal{A}=q_1〈p_1,\dots ,q_n〉p_n,\hfill \\ {\eta }_j=\pi \left(a,q_j\right),j\in \left[1,n\right]\hfill \\ \pi \left(a^{\prime },\underset{grouping}{\underbrace{\left[\left[s_i\right]\right]a\left[1\right]〈r_i,\dots ,\left[\left[s_j\right]\right]a\left[1\right]〉r_j}}\right)\text{otherwise, let}{a}^{\prime }=\mathsf{group}\left(a,0̸,\mathcal{A},\mathcal{G}\right)\hfill \\ \mathcal{G}=s_1〈r_1,\dots ,s_m〉r_m,\hfill \\ \sigma =\left\{s_i,\dots ,s_j\right\},1\le i\le j\le m\hfill \end{array}\right.}$

Example: semantics To illustrate the semantics of the group operator, we consider the alternative version of the code shown at Line 12 (and presented as a second example at the beginning of this section), where we want to aggregate for $\underset{¯}{quality}$ but we also want to keep those values grouped by $\underset{¯}{year}$, $\underset{¯}{month}$, and $\underset{¯}{day}$.

 
 
 
   ______________________________________________________________________________________________________ 
 
   sl ← γ ( sl , ( quality  ) , ( day  , month  , year  ) )    

In the semantics, the first thing we do is the de-sugaring of paths—namely $\underset{¯}{quality}$, $\underset{¯}{day}$, $\underset{¯}{month}$, and $\underset{¯}{year}$, which respectively expand to $\underset{¯}{quality}〈〉\underset{¯}{quality}$, $\underset{¯}{day}〈〉\underset{¯}{day}$, $\underset{¯}{month}〈〉\underset{¯}{month}$, and $\underset{¯}{year}〈〉\underset{¯}{year}$—and then we apply the de-sugared group operator on — sl— (which, we remind, contains only values corresponding to the dates 2020-11-29 and 2020-11-30, represented by the trees $t_{29}^1,t_{29}^2,\dots$ below).

The lookup operator

λ(a, q, a′, r, p)

The purpose of the lookup operator is to join the trees in a source array a with the trees in an adjunct array a′. For those values obtained by applying the path q on a, the lookup operator pairs them with the equivalent values obtained by applying r on the adjunct array a′ and it projects the latter under path p in the paired trees of a.

Example Before commenting on the application of the lookup in Listing 2, we describe the results of the group at Line 4 and of the two projections, respectively at Line 5 and Line 13. At Line 4, we aggregate the temperatures in the — tmp— data structure, which results into

 
 
 
  __________________________________________________________________________________________________________ 
 
  [ υ { t : [ 36 { } , 36 { } , 37 { } ] } ]    

The projection at Line 5 performs two actions over the — tmp— data structure. First, it keeps only the node t (holding the temperatures filtered for the days of interest). Second, it projects into the filtered data structure the pseudo-identifier (— pseudoID—) under the node patient_id.

 
 
 
  ________________________________________________________________________________________________________________________ 
 
  [ υ { t : [ 36 { } , 36 { } , 37 { } ] , patient_id : [ ' id_xxx ' { } ] } ]    

The projection at Line 13, similar to the one above, keeps only the node quality (holding the quality of the sleep for the days of interest) and it projects the — pseudoID— under the node patient_id.

 
 
 
  _____________________________________________________________________________________________________________________ 
 
  [ υ { quality : [ ' good ' { } , ' good ' { } , ' poor ' { } , ' good ' { } ] , 
   patient_id : [ ' id_xxx ' { } ] } ]    

We can now comment on the lookup at Line 14, which we report below for convenience.

 
 
 
  _______________________________________________________________________ 
 
  14 bs ← λ ( sl , patient id  , tmp , patient id  , temperatures  )    

The instruction joins the data structures — tmp— and — sl— by pairing the values under the path $\underset{¯}{patient\text{\_}id}$ (this is a special case where the left and right paths of the join coincide, i.e., the path $\underset{¯}{patient\text{\_}id}$). The last path in the application, i.e., $\underset{¯}{temperatures}$, indicates where the values from the right data structure (— tmp—) should be projected in the paired values of the left one (— sl—).

At Line 14, we store the result of the application of the lookup into a new variable — bs— (standing for bio-signals).

 
 
 
  ____________________________________________________________________________________________________________________________________ 
 
  υ { quality : [ ' good ' { } , ' good ' { } , ' poor ' { } , ' good ' { } ] , 
   temperatures : [ υ { t : [ 36 { } , 36 { } , 37 { } ] , 
   patient_id : [   ' id_xxx ' { } ] } ] , 
   patient_id : [   ' id_xxx ' { } ] }    

For completeness, we report the result of the last step of Listing 2, at Line 15, where we apply the project operator to reshape the data structure for the invocation of the — detectEncephalopathy— functionality at Line 16.

 
 
 
  _________________________________________________________________________________________________________________________ 
 
  υ { quality : [ ' good ' { } , ' good ' { } , ' poor ' { } , ' good ' { } ] , 
   temperatures : [ 36 { } , 36 { } , 37 { } ] , 
   patient_id : [   ' id_xxx ' { } ] }    

Semantics In the semantics of the lookup, for each element a[i] (1 ≤ i ≤ #a), we use the tree version of the project operator (π(t, Π), cf. ‘The project operator’) to merge the element a[i] with the paired values from a′ under r. Since, by its definition, π(t, Π) corresponds to the merging (⊕) of the single applications of each component in the sequence Π, we use this to merge the source tree a[i] with the paired elements in a′. Hence, for each element a[i], we define Π_i as the sequence ɛ, μ(a′, φ_i)〈〉p. The projection for the first component (ɛ) returns the original tree (a[i]). The projection for the second component (μ(a′, φ_i)〈〉p) injects the result of the match μ(a′, φ_i) into the path p, where the criterion φ_i, equal to r = [[q]]^a[i], selects those values in a′ that under r coincide with the array found under q in a[i].

Note that when for some i we have q not present in a[i] (i.e., [[q]]^a[i] = α) the lookup operator joins a[i] with those trees in a′ where r does not exist (i.e., μ(a′, r = α)). ${\displaystyle \lambda \left(a,q,a^{\prime },r,p\right)=\text{[}\pi \left(a\left[1\right],{\Pi }_1\right)\text{]}::\cdots ::\text{[}\pi \left(a\left[\mathtt{\#}a\right],{\Pi }_{\mathtt{\#}a}\right)\text{]}\text{where}\left\{\begin{array}{c}1\le i\le \mathtt{\#}a\hfill \\ {\Pi }_i=\varepsilon ,\mu \left(a^{\prime },{\phi }_i\right)〈〉p\hfill \\ {\phi }_i=\left(r={\left[\left[q\right]\right]}^{a\left[i\right]}\right)\hfill \end{array}\right.}$

Example: semantics Below, we report the unfolding of the execution of the lookup at Line 14. Since we have one value in — sl—, we do not perform a concatenation of arrays but we just apply the projection for — sl—[1]. In the three reductions below, first, we retrieve the content of ${\left[\left[\underset{¯}{patient\text{\_}id}\right]\right]}^{\mathtt{sl}\left[1\right]}$, then, we execute the match (which essentially returns the whole content of the — tmp— variable), and, finally, we merge — sl—[1] (obtained by the projection under ɛ) with the result of the match projected under path $\underset{¯}{temperatures}$.

 
 
 
  _________________________________________________________________________________________________________________________ 
 
  [ π( sl [1],(ɛ,μ( tmp ,patient id = [[ patient id ]]sl [1] ) 〈〉 temperatures  )) ] 
    ⇒ [ π( sl [1],(ɛ,μ( tmp ,patient id = [ ' id_xxx ' { } ] ) 〈〉 temperatures  )) ] 
    ⇒ [ π( sl [1],(ɛ, [ υ { t : [ 36 { } , 36 { } , 37 { } ] , patient_id : [ ' id_xxx ' { } ] } ] 
    〈〉 temperatures  )) ] 
    ⇒ [ υ { quality : [ ' good ' { } , ' good ' { } , ' poor ' { } , ' good ' { } ] , 
    patient_id : [ ' id_xxx ' { } ] } ⊕ υ { temperatures : [ υ { t : [ 36 { } , 36 { 
   } , 37 { } ] , 
    patient_id : [ ' id_xxx ' { } ] } ] } ]    

Extending Jolie/Tquery with query pipelines

Besides providing a faithful implementation of Tquery, we decided to extend Jolie/Tquery to support multi-stage queries both for reasons of performance and familiarity with the MongoDB Aggregation framework (MongoDB Inc., 2022).

The extension is minimal and provides an interesting point for showcasing the flexibility of the Jolie language in evolving existing projects.

Namely, the extension regards the API and the behaviour. We report in Listing 5 the changes to the Jolie/Tquery API and we omit, as done above, to present the Java code of the implementation, which is a straightforward sequentialisation of calls to the other implemented operators.

In the API, we add the — pipeline— operation among the operations in the — TqueryInterface— — interface—. The new operation requires an associated request — type— that contains the specification of the multi-stage queries. Having defined the — type—s of the other operations as independent components comes in handy. Indeed, the — Pipeline— — type— defines its multi-stage query as an array (under the sub-node — pipeline—) of subtrees specified through the — type—s of the other operations. For instance, at Line 11 in Listing 5, a — match— (— Query—) stage has the structure of the φ — type—, which is also the one used by the — match— operation (in the

 
Type    

 
 
 
   _____________________________________________________________________________ 
 
   1 interface  TqueryInterface  { 
   2  RequestResponse : 
  3  match  ( μ Type  ) ( QueryResponse  ) , 
  4   // ... 
  5  pipeline ( Pipeline  ) ( QueryResponse  ) 
  6 } 
   7 
      8 type  Pipeline : { 
   9  data * : undefined 
  10  pipeline [ 1 , * ] : 
  11  { matchQuery  : φ } 
   12   |  { projectQuery [ 1 , * ] : Π } 
   13   |  { unwindQuery  : Path  } 
   14   |  { groupQuery  : Group_Exp  } 
   15   |  { lookupQuery  : { 
   16  leftPath  : Path 
  17  rightData * : undefined 
  18  rightPath  : Path 
  19  dstPath  : Path 
  20  } 
   21  } 
   22 } 
         Listing 4: Pipeline support extension (fragments).    

The curious reader could wonder why we did not specify the whole Jolie/Tquery interface through the single — pipeline— operation. Our point is that, by having both possibilities, users can opt for the modality that best suits their scenario. For instance, when developing and debugging a query, it is useful to look at the shape of the single invocations and responses. Moreover, while pipelines help to make local sequential invocations efficient, they make the code harder to distribute, since the query now lives as an indivisible data structure. On the contrary, if we found out that a specific stage of a query, e.g., the match at Line 3 or the unwind at Line 9 of Listing 2, would benefit from scaling it over multiple copies, we could do that by isolating each operation into a dedicated service and redirecting their inputs/outputs to perform our original local query as a distributed one. In that case, despite the architectural change, the logic of the query would remain intact.