I-Cubid: a nonlinear cubic graph-based approach to visualize and in-depth browse Flickr image results

Limitations

The utilization of images is experiencing exponential growth in the present era (Bouchakwa, Ayadi & Amous, 2020). The web image search results representation over SERPs is primarily linear, where results are ranked in lists by considering the textual similarity of query and retrieved web image results (Al-Jubouri, 2019). The SERPs usually present the linearly organized web image search results via primitive grid layouts, typically 2-dimensional layouts (Alzubi, Amira & Ramzan, 2015). The in-depth browsing of image results is not possible (Kopalle et al., 2022). Recently research has been focusing on image search result organization as a viable approach to enhance the quality of image retrieval on the web (Tekli, 2022). The traditional organization of web image search results in 2-dimensional grids allows horizontal-vertical scanning of the ranked image results. However, the results at lower-ranking positions could be more challenging to reach and navigate (Rashid, Saleem & Ahmed, 2021). The complexity associated with advanced approaches makes them cumbersome to instantiate over the real dataset of web image search results since the graphs are dense (Rashid, Saleem & Ahmed, 2021). Navigating the desired results in the dense graph could be less convenient. The linear adaptation of SERPs permeates globally to provide efficacy in exploring web image search results. However, the focus is not to give the in-depth browsing of web image search results organized in traditional grid layouts.

Motivations

The in-depth browsing that allows the users to navigate non-linearly within the web image search results presented in standard grid layouts inspired us to conduct this research. We want to organize the web image search results retrieved from web sources in cubic graphs and enable their exploration via a particular SUI design; so the users can perform in-depth browsing along with traditional horizontal-vertical scanning in an interactive, integrated, and usable way. We are particularly interested in reaching non-linearly web image search results at lower-ranking positions. To do this, in this research, we propose the I-Cubid approach. In particular, by the I-Cubid approach, it would be possible to: (i) retrieve, visualize, and in-depth browse the retrieved web image search results from the online web source, i.e., Flickr, and (ii) explore a cubic graph space of web image search results via various components of a comprehensive SUI design.

Research questions

The linear ordering of web image search results presented in traditional grid layouts may not support in-depth browsing, navigation within the search results, and reachability. This research focuses on representing the web image search results in cubic graphs and enabling their exploration via a particular SUI design. We formalized the following research questions:

• RQ1: How can a cubic graph and associated SUI design be exploited mainly in the ranked representation of web image search results to support visualization and in-depth browsing activities?

• RQ2: Does a cubic graph result space instantiated over web image search results provide visualization, in-depth browsing, and address reachability issues in an integrated way?

• RQ3: Is the interaction with web image search results in visualization and in-depth browsing provided via the I-Cubid approach usable and satisfying the users’ information needs?

• RQ4: Do users prefer the interaction as visualization and in-depth browsing via cubes in the I-Cubid approach over the traditional 2-D grid-based layout in web image results exploration?

The readiness of advanced exploration options to integrate into exploration requires further investigation. The research questions discussed in this section addressed the particular research directions. Notable, RQ1 addresses the formation of a cubic graph and SUI design; RQ2 describes the usage of instantiated results space and interaction with it by the users; RQ3 reveals the usability and its associated dimensions; RQ4 provides the comparison of usability related to the proposed SUI design and standard grid layouts.

Contributions

Our approach used cubic graph representation to visualize web image search results. The users can browse the web image results in a usable way, view details associated with image results, and access images and their associated information from their sources in an integrated manner. The users can navigate horizontally-vertically and across the planes (in-depth) dynamically to explore the web image search results in a usable way. Our approach enables the nonlinear exploration possibilities to interact with the cubic graph of image results. The approach is further compared with traditional grid-based layouts. The cubic visualization gives satisfactory mechanisms to explore the web image search results. The exploration of results via cubes passed various usability tests and was comparatively more convenient than traditional grid layouts in different usability dimensions.

Following is the organization of this paper. In Section 2, we discuss the literature review. In Section 3, we describe the I-Cubid: approach. Section 4 discusses the research methods employed to validate the I-Cubid approach. In Section 5, we analyze the experimental results. Finally, in Section 6, we conclude our discussion and highlight future research.

Background

The retrieval techniques provide access to web image repositories by exploiting the textual and visual information modalities associated with them (Unar et al., 2019; Zha et al., 2020). The former as keywords with web images as textual descriptions, annotations, tags, meta information, etc., and later as low-level visual features extracted from them (Datta et al., 2017; Saddal, Rashid & Khattak, 2019). However, existing retrieval techniques usually employ textual modalities to explore the web image search results. The search paradigm consequently satisfies information needs by exploring image results retrieved by considering queries, search intents, and relevance feedback (Wu et al., 2019a). The satisfaction of information needs via the visual search paradigms is linked to information needs expression and browsing of the retrieved results via SERPs (Zhang et al., 2018; Wu et al., 2019b). Over the years, linear and nonlinear exploration layouts emerged as separate paradigms. In the following, we discuss linear, nonlinear, and state-of-the-art image results exploration research.

Linear exploration

Traditionally web image search results interaction usually employs linear interaction layouts to provide exploration, where users can give textual query terms (usually via bi-grams and tri-grams) or visual words (image examples) (Jégou, Douze & Schmid, 2010). Consequently, interaction with search results via grid layouts is possible, which generally supports exploration via lookup-based activities (Vrochidis et al., 2019). In this case, the support for the exploratory activities could be minimal. The only option is to interact with the image results via 2-dimensional scanning of grid-based layouts, where the users can traverse results horizontally-vertically (Xie, 2019; Xie et al., 2019). The grids may consider the similar relationships between the query and textual information associated with web image search results. Notably, most grid layouts do not consider the similarity and semantic relationships in the image results. However, the grids usually represent the ranking order, which may decrease from top-bottom and left–right. The interaction options are restricted due to the limitations associated with them. Figure 1 elaborates on exploring web image search results via a linear interaction paradigm. As it emerges from Fig. 1 that the exploration is possible via grid layouts, image thumbnails, facets, color schemes, and cluster tags in web image search engines. However, they provide 2-D grid plane exploration. The plane represents images as a single-layer 2-D grid consisting of rows and columns. Each plane contains a set of highly similar images. Further in-depth browsing of a selected image plane is not possible.

Nonlinear exploration

The nonlinear exploration approaches usually organize the web image search results in nonlinear data models, e.g., graphs, trees, multi-lists, knowledge networks, self-organizing maps, etc., by exploiting similarity and semantic relationships. The nonlinear representation is further exploited in exploring web image search results. The focus is to provide the visualization of results that enhance the users’ exploration activities to seek the relevant images. The nonlinear exploration facilitates the exploratory search tasks (Vrochidis et al., 2019). The SUI is usually based on the underlying nonlinear organization of the web image search results. The user can navigate results non-linearly and perform exploration activities, including navigation, in-depth browsing, visualization, and access to the desired results (Rashid, Saleem & Ahmed, 2021). However, exploring web image search results via dense and nonlinear data models is challenging via the interactive exploration tools (Yan et al., 2017). The mechanism of the advanced interactions exploits the dynamic grid layouts, clusters, heat-maps, etc., in the nonlinear exploration over Picsbuffet (Jung et al., 2021), Google Image Swirl (Jing et al., 2012), t-SNE algorithm (Wattenberg, Viégas & Johnson, 2016), and interactive graph (Rashid, Saleem & Ahmed, 2021).

State-of-the-art

Approach

The proposed I-Cubid approach exploits a particular nonlinear ranked representation to explore the web image search results retrieved from Flickr in real time against the user-issued queries. While the proposed approach can be adapted to any image dataset, we chose Flickr as a baseline for realistic results comparability. It provides exploration via a particular 3-D cubic structure and interaction with the web image search results in various ways. In the I-Cubid approach, users can express their visual information needs by introducing simple textual query terms (usually bi-grams and tri-grams) and interact with web image search results via multiple integrated layouts. However, the approach primarily focuses the nonlinear navigation as in-depth browsing within the retrieved web image search results. The objective is to enable the nonlinear exploration of web image search results in a usable way. Figure 2 illustrates the I-Cubid approach.

The I-Cubid approach enables the visual information need expression via an interactive SUI. The users can formulate the textual queries for visual search by giving the multiple keywords connected via Boolean (AND, OR, NOT) operators (Fig. 2A). The keywords are initially transferred to a meta-search engine, which pre-processes textual query terms (performs necessary text indexing operations, i.e., stop-word removal and stemming) and dispatches the user query terms to a selected web image search engine (Fig. 2B). The results are retrieved from the Flickr database via indexes of web search engines since Flickr supports the textual indexing and retrieval of images (Fig. 2C). The results retrieved against the textual query terms from Flickr are initially stored in a web image results space, where the association within the results is unknown (Fig. 2D). Initially, our approach ranks web image search results in a 2-D grid layout (Fig. 2E). The I-Cubid approach employs a particular algorithm to compute inter-image similarity ranking to represent the results in a 3-D cubic data model (Fig. 2F).

The I-Cubid approach represents the web image search results in separate 2-D planes. Finally, it assembles them in a 3-D cubic representation. It is further utilized as a results space for the visualization and in-depth browsing of Flickr images retrieved against textural query terms (Fig. 2G). In the I-cupid approach, the users can explore the web image search results via a navigable and dynamic 3-D cubic visualization scheme. The users can initiate image search results-seeking activities by traversing (top-down and left–right) 2-D grids and in-depth browsing of 3-D cube planes of web image search results. Initially, the retrieved web image search results are ranked in a grid-based layout by considering the textual similarity of the query and textual information associated with the retrieved image results since our approach also retrieves the textual information associated with web image search results (Fig. 2C).

The I-Cubid approach employs the inter-image textual similarity ranking algorithm and cube modeling algorithms to transform the web image search results represented via 2-D grid-based layouts into multiple 2-D planes and 3-D cubic representations. The I-Cubid approach visualizes 3-D cubic representations to the users via an interactive SUI design. I-Cubid mainly provides the visualization and in-depth browsing of Flickr images by employing a 3-D cube and SUI design; the former represents the web image search results retrieved from the Flickr database against textual query terms and later visualizes them in a variety of integrated and usable ways. We will discuss the 3-D cube formalization, SUI design, and usage scenario in the following.

Cubic graph formation

Let I be the web image search result space retrieved against a user-based query formulation. The I = i₁, i₂, …, i_n shows there is n number of retrieved web images comprising textual metadata $\mathcal{D}=\left(...\right)$. The I is visualized in grid structure $\mathcal{G}$ and cubic structure making it a cubic graph data model I_CG to enable exploration activities (Fig. 3). Formally a graph model is represented as an acyclic, labeled graph G = (V, E) and I_CG = G therefore, I_CG = (V, E), where V is image nodes and E is the non-negative weighted edges where $E=\left\{e_1,e_2,\dots ,e_n\right\}$ and $V=\left\{v_1,v_2,\dots ,v_m\right\}$.

The E counts the similarity relationships between V as the weights. It means edges display the similarity between images. The I_CG have multiple planes P which forms a cube shape where $P=\left\{p_1,p_2,\dots ,p_n\right\}$, shows n number of planes depending upon the I (Figs. 3A, 3B). The maximum plane P_max should P_max ≤ n and the maximum number of planes visualized P_vmax at a time P_vmax = 5. Each P in I_CG have its neighbor image nodes $I_{ne}=\left\{i_{ne1},i_{ne2},\dots ,i_{nen}\right\}$ where I_ne ∈ I and I_ne ∈ P⋀ P ∈ I_CG∴I_ne ∈ I_CG.

The arrangement of I depends upon E in I_CG; consequently, I_ne are generated. The position of I_ne in P also depends upon E in I_CG and ∀i_n∃e_n. The degree of an image node deg(n) is the number of adjacent edges presenting the I_ne and ∀e_n the degree range is 3 ≤ deg(n) ≤ 6 to form I_CG. Keeping in view, the following case properties hold:

• In the case outer corner image nodes of p₁⋀p_n, ∀e_n ∈ I_CG the $deg\left(n_i\right)=3\to I_{ne}=3$ (Fig. 3C).

• In the case of outer image nodes except corners of p₁⋀p_n, ∀e_n ∈ I_CG the $deg\left(n_i\right)=4\to I_{ne}=4$ (Fig. 3D).

• In the case of inner image nodes of p₁⋀p_n, ∀e_n ∈ I_CG the $deg\left(n_i\right)=5\to I_{ne}=5$ (Fig. 3E).

• In the case of outer corner image nodes of p₂, …, p_n−1, ∀e_n ∈ I_CG the $deg\left(n_i\right)=4\to I_{ne}=4$ (Fig. 3F).

• In the case of outer image nodes except corners of p₂, …, p_n−1, ∀e_n ∈ I_CG the $deg\left(n_i\right)=5\to I_{ne}=5$ (Fig. 3G).

• In the case of inner image nodes p₂, …, p_n−1:∀e_n ∈ I_CG the $deg\left(n_i\right)=6\to I_{ne}=6$ (Fig. 3H).

The I_ne are created on the based of $deg\left(n_i\right)$ according to the properties to generate P forming I_CG. The E is created if the edge weight W exceeds the threshold γ. The details of thresholds, number of planes, and similarity measures are discussed in the other sections.

3-D cubic in-depth browsing and navigation

The I-Cubid graph data model initially represents web image search results into 2-D planes and further generates 3-D cubic representation. Moreover, it provides a web image search results space visualization and in-depth browsing of Flick images. The users initiate exploration activities via navigable and dynamic 3-D cubic web image search result arrangement. Users commence their image-seeking activities by in-depth browsing of 3-D cube planes. The web image search results-based 3-D cubic representation comprises multiple planes, and the browsing pattern involves the image’s traversing and plane’s switching (Fig. 4). The image position can be different in the same plane and can navigate to other planes to browse deeply in web image search results. Figure 4 navigation path shows that browsing initially begins with the first plane p₁ center position, then shifts upwards and moves to the second plane p₂ similar to the other planes till the last plane p_n.

SUI design

This section elaborates on an SUI design for cubic visualization and exploration. The SUI design consists of multiple panels providing interaction with Flickr images. The main objective of SUI design is the cubic visualization of the Flickr images to provide in-depth browsing and navigation in the cube exploiting the image relationships. Figure 5 depicts an SUI design comprising multiple panels. The details of the SUI design’s panels are as follows.

Query panel (QP): The QP allows users to fulfill their information needs by formulating textual queries (Fig. 5A). The textual queries enable visual search by giving multiple keywords via Boolean keywords (AND, OR, NOT). The simple textual query terms are usually bi-grams or tri-grams connected via Boolean keywords fulfilling the users’ exploration needs. These Boolean keywords activate the Boolean operations between the query keywords to filter the image retrieval result space. In this case, the user issues the query “cat” in the QP, which is dispatched to the Flickr database in real time.

Grid view panel (GVP): The GVP shows the query-based image search results from Flickr in a standard 2-D grid layout (Fig. 5B). The clickable image thumbnails maintain their original aspect ratio leading toward textual information. Initially, the images retrieved against the “cat” query in the 2-D grid utilize the Flickr ranking. The checkbox on the GVP enables Jaccard-based ranking, computed by relevancy between query keywords and image meta-tags. It shows the number of relevant images found on Flickr as well. The GVP allows to transverse the image horizontally-vertically in a plane and navigate to the cubic browsing panel via the cube icon. During the exploration, the user selects the most appealing “cat” image in the initial to further explore the related images (Fig. 4 (P₁)).

Grid image view panel (GIVP): The GIVP displays an image with its textual metadata (Fig. 5C). After a user selects an interesting “cat” image, it presents a chosen image thumbnail in the original aspect ratio. The meta-information provides views, favorites, and comments on Flickr. It elaborates on the selected image format (Fig. 4 (P₁)) and the date and time of the image taken. The GIVP also provides an option to download the image.

Cubic browsing panel (CBP): The CBP visualizes the images in a 3-D cube layout (Fig. 5D). The 3-D image cube comprises multiple planes (Fig. 4 (P₁ ... P_n)), and each plane contains 25 distinct “cat” images related to the clicked “cat” image. The image thumbnails having a fixed aspect ratio are presented in each plane. The CBP allows browsing within the multiple cubic planes to enable in-depth exploration. Unlike traditional grid exploration, users can view any cubic plane via the ‘switch plane’ option to further explore the related images. Initially, the cube images use Flickr-based ranking, and the checkbox ranks the cube images via Jaccard similarity. The selection of image display in planes is based on Jaccard similarity ranking and descending order sorting. We chose Jaccard similarity instead of embeddings, such as Glov (Pennington, Socher & Manning, 2014; Mikolov et al., 2013), due to its least computationally requirements and to avoid sparsity problems connected to a vector-based representation (Rashid & Bhatti, 2017). Each cube shows a fixed number of planes, i.e., five cubic planes of the selected “cat” image, at a time. In cases where the number of images exceeds the five planes, users can navigate to ‘next’ to view more planes of the same cube.

Cubic image view panel (CIVP): The CIVP exhibits the selected image thumbnails in the original aspect ratio (Fig. 5E). It shows the image’s metadata, including the number of views, favorites, comments, and image creation details, i.e., time and date. It also specifies the image format and allows to download of that particular image. The CIVP enables the view of the neighboring “cat” images related to the selected “cat” image (Fig. 3).

Cubic navigation panel (CNP): The CNP enables the navigation in the neighboring images of select “cat” image (Fig. 5F). The neighbor image includes the images from the same plane, e.g., in the case of the second plane as shown in (Fig. 4 (P₂)), the neighbor includes left, right, top, bottom, and the images from deeper planes, e.g., front and back images (Fig. 4 (P₁ and P₃)). Hence, the neighbors are computed based on the arrangement in the 3-D cube layout.

I-Cubid interaction mechanism with SUI design

The I-Cubid interaction enables users to express information needs, instantiate exploration, 2-D nonlinear navigation, and 3-D nonlinear navigation via SUI panels (Fig. 6).

Expressing information needs: The users express their information needs via keywords-based queries. The QP allows keyword submission to initiate the exploration activities. The QP is directly connected to other SUI panels (Fig. 6A).

Instantiating exploration: The users can perform traditional 2-D grid-based browsing in GVP. The image result space depends upon dual rankings (Flickr-based and I-Cubid based), enabling exploration (Figs. 6B, 6K). The 2-D grid-based exploration leads towards associative panels, i.e., GIVP and CBP, to enable 2-D and 3-D nonlinear navigation (Figs. 6L, 6D). The objective is to give users an overview of the Flickr-based retrieved image result space-providing exploration activities.

2-D nonlinear navigation: The users can navigate non-linearly in the traditional 2-D grid via GVP. It allows one to view selective image thumbnails and browse within the GIVP (Fig. 6L). It will enable the access of web image search results non-linearly and horizontally-vertically traversal. The associative details are displayed on GIVP, which further navigates to the actual web image source on Flickr (Fig. 6E). The web image search result space arrangement depends upon dual rankings (Flickr-based and I-Cubid-based), enabling nonlinear navigation. The 2-D navigation of GVP and GIVP leads towards CBP enabling 3-D nonlinear navigation.

3-D nonlinear navigation: The CBP provides 3-D navigation and in-depth browsing. The CBP shows the image result space in a 3-D cube structure allowing users to navigate within the cube and its multiple planes. The cube provides also provides a dual ranking and allows one to navigate back to GVP (Figs. 6J, 6D). The CBP is also connected with other cube panels like CIVP and CNP (Figs. 6F, 6H). The users can browse associative image details on CIVP, which further navigates to the actual Flickr source (Fig. 6G). The cube structure consists of multiple planes that allow 3-D navigation via CNP (Fig. 6H). In CNP, each cube plane is connected with other planes showing retrieved images to activate in-depth browsing, and each image highlights its neighboring images (left, right, top, bottom, front, and back). The users can even navigate back to the CBP, enabling in-depth browsing and exploration (Fig. 6I).

Implementation details

We implemented I-Cubid as a web application. The backend was developed using the PHP scripting language suited to web development. The backend contained the logic to retrieve the image results from the Flickr publicly available API in real-time against user queries and process it into the nonlinear cubic graph. The front end was developed using HyperText Markup Language (HTML), Cascading Style Sheet (CSS), and JavaScript to provide the interaction with the search user interface. We used BootStrap (https://getbootstrap.com/) and Vue.js (https://vuejs.org/) libraries to make the search user interface responsive across various screen sizes. The I-Cubid project was managed on the Visual Studio (https://visualstudio.microsoft.com/) integrated development environment. The website was hosted on the Apache XAMPP (https://www.apachefriends.org/) server. From the performance perspective, the proposed tool, on average, took 5 s from query dispatching to search results retrieval on the search user interface. The further hardware details are discussed in section 4.1.

Apparatus & dataset

The I-Cubid tool was installed on a lab computer with Windows 10 Pro operating system, 10 GB RAM, and a 3 GHz dual-core processor. Access to the computer was provided with a wired mouse and keyboard. The screen recording software was also installed on the computer to record users’ interactions with the tool. In this study, we employed a real dataset of Flickr images retrieved via simple textual query terms to instantiate the I-Cubid interaction.

Users

We recruited 18 users for the usability evaluation of the I-Cubid tool. According to existing literature, a minimum of five users are required to uncover 80% of the usability issues, and 12 users are adequate for obtaining statistically significant results (Alroobaea & Mayhew, 2014). To make the evaluation process more rigorous, we recruited users from diverse backgrounds (housewives, students, and professionals). Among them, 10 were males, and eight were females. The users’ minimum, maximum, and mean ages were 21, 45, and 28 years, respectively. The users’ complete details are provided in Table A1.

Study design & instruments

We employed within-subjects usability evaluation. The users were asked to evaluate both systems; baseline and I-Cubid. The baseline system provided a grid-based image presentation, and I-Cubid provided a 3D cube-based image presentation. The users were initially given a demographic survey and brief instructions for evaluating and simulating the tools. This introductory session consisted of 5 to 10 min. Afterward, the users were asked to choose a topic of interest from four available issues. Finally, the Questionnaire for User Interface Satisfaction (QUIS) (Chin, Diehl & Norman, 1988) and System Usability Scale (SUS) (Brooke, 1996) instruments were deployed to measure the system usability and user interface satisfaction on both systems. The affiliated university board duly approved this study design (Ethical Application Ref: UR \BSCS \Projects \0039). Furthermore, verbal consent from the participants was taken before the conduct of human studies (Table A2).

Topics & Tasks

We designed four generic topics; “Gold Fish in Aquariums”, “Bullet Trains in Japan”, “Fighter Aircraft in Pakistan”, and “Indian Tiger”. These exploration topics were designed for usability evaluation. Each topic consisted of a guided task with complete procedural instructions provided to the user and an unguided task without guided instructions. The guided task aimed to increase users’ familiarity with the tool to reduce the learning effect. The tasks were designed to evaluate the grid view panel and cubic panel. Tasks were performed by users using the topic of their own choice. The topics and tasks details are displayed in Table A3.

System usability

The SUS administered to the users shows an overall average score of 80.8 on the grid-based image presentation. According to the ranking yardstick presented in Lewis & Sauro (2018), the grid presentation achieves a grade ‘A’ in the usability test and falls into the top 90–95 percentile of the best usable tool. On the I-Cubid, the system usability administered to the users reported an overall average score of 78.2. According to the ranking yardstick presented in Lewis & Sauro (2018), it achieves a grade of ‘B+’ and falls into the top 80–84% of the usable performing tools. The overall SUS results for the cube and grid-based system, including T-Test, distribution frequency (df), mean differences, standard deviation, and error mean, are displayed in Table 1.

User interface satisfaction

The QUIS scale measures five categorical aspects via several questions about the administered tool. These categories measure (1) quality of interface design (such as readability, organization, and visual aids, etc.), (2) terminology and system information (including ease of use and user-friendliness), (3) user learnability, (4) quality of developed tool in terms of speed, accuracy, and usefulness, and finally (5) overall perceived reaction to the tool. This scale includes a total of 27 questions to measure all the categorical aspects of the tool.

The users on the grid-based interface reported an overall average of 83% on the QUIS. Therefore, grid presentation is perceived highly satisfactory among the users in terms of the usability and interface satisfaction. For the user interface satisfaction, the cube-based images presentation overall achieves an average of 83.2%. The overall QUIS results for the cude and grib-based system are displayed in Table A5 and Table A7, respectively.