Lora Based Remote Patient Monitor

Remote Sensing Change Interpretation And Analysis

Abstract—Monitoring changes in the Earth’s surface is crucial for understanding natural processes and human impacts, neces- sitating precise and comprehensive interpretation methodologies.

Remote sensing satellite imagery offers a unique perspective for monitoring these changes, leading to the emergence of remote sensing image change interpretation (RSICI) as a significant research focus. Current RSICI technology encompasses change detection and change captioning, each with its limitations in pro- viding comprehensive interpretation. To address this, we propose an interactive Change-Agent, which can follow user instructions to achieve comprehensive change interpretation and insightful analysis, such as change detection and change captioning, change object counting, change cause analysis, etc. The Change-Agent integrates a multi-level change interpretation (MCI) model as the eyes and a large language model (LLM) as the brain.

The MCI model contains two branches of pixel-level change detection and semantic-level change captioning, in which the BI- temporal Iterative Interaction (BI3) layer is proposed to enhance the model’s discriminative feature representation capabilities. To support the training of the MCI model, we build the LEVIR- MCI dataset with a large number of change masks and captions of changes. Experiments demonstrate the SOTA performance of the MCI model in achieving both change detection and change description simultaneously, and highlight the promising application value of our Change-Agent in facilitating compre- hensive interpretation of surface changes, which opens up a new avenue for intelligent remote sensing applications. To facilitate future research, we will make our dataset and codebase publicly available at https://github.com/Chen-Yang-Liu/Change-Agent Index Terms—Interactive Change-Agent, change captioning, change detection, multi-task learning, large language model.

T

HE changes in the Earth’s surface, as a dynamic indicator of the Earth’s system, profoundly affect the planet’s evolution and the survival of humankind. Observing and analyzing these changes is pivotal for advancing sustainable human development . Remote sensing (RS) satellite im- agery, offering a unique “God’s perspective,” emerges as an effective tool for monitoring Earth’s dynamic changes. Remote sensing image change interpretation (RSICI) has emerged as Chenyang Liu, Keyan Chen, Haotian Zhang, Zipeng Qi, and Zhenwei Shi are with the Image Processing Center, School of Astronautics, with the Beijing Key Laboratory of Digital Media, and with the State Key Laboratory of Virtual also with the Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China.

Chenyang Liu is also with Shen Yuan Honors College of Beihang Univer- sity, Beijing 100191, China. with Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China.

a significant research focus . RSICI aims to identify and analyze changes from images captured at different times in the same geographical area. It can provide decision-making support for environmental protection , urban planning , resource management , etc.

Current RSICI technology primarily encompasses change detection and change captioning. Change detection accurately localizes the spatial location of surface changes at the pixel level , . In contrast, change captioning aims at articu- lating the attributes and meanings of changes using natural language, emphasizing the semantic-level understanding of changes –. Despite significant advancements in both areas, their respective limitations prevent us from obtaining comprehensive change interpretation information by utilizing a single technology. Specifically, although change detection can accurately localize the changed area, it lacks a deeper semantic-level understanding of changes’ underlying mean- ings, such as the characteristics of ground objects (e.g., cat- egory, color, shape), the spatial relationships between them (e.g., “beside”, “around”) and dynamics of changes (e.g., “appear”, “removed”). Conversely, change captioning can pro- vide rich semantic-level interpretation information but may fall short in providing precise pixel-level change localization.

Therefore, there is a pressing need to explore a multi-level change interpretation (MCI) approach providing both pixel- level and semantic-level change information, facilitating pre- cise change localization while delving into the essence and implications of the changes.

Moreover, practical applications often require comprehen- sive analysis and further processing of both pixel-level and semantic-level change interpretation results to meet specific needs. For instance, users may require statistical analysis of changed object numbers, consuming significant time and effort from researchers and demanding users’ technical proficiency.

To address these challenges, we propose an interactive Change-Agent based on a novel MCI model and a large language model (LLM). Fig.1 illustrates the advancements achieved by our Change-Agent compared to previous tech- nologies. Different from the previous single technology, with the help of the MCI model, our Change-Agent can simul- taneously achieve precise pixel-level change detection and semantic-level change captioning, thereby providing users with comprehensive change interpretation information. Notably, our Change-Agent boasts interactive capabilities, enabling users to communicate their queries or requirements regarding sur- face changes. Leveraging the MCI model and the LLM, the

A) Previous Vs. Ours

Some houses are built on both sides of the road. Specifically, how many houses and roads have changed? 9 houses and a road have changed.

In ‘D:/Map.Png’

The road change map have been saved.

Built On Both

sides of the road.

B U I L T O N B O T H

sides of the road. Fig. 1. The comparison between previous single technology and our Change-Agent. Our Change-Agent can simultaneously achieve precise pixel-level change detection and semantic-level change captioning. Besides, it boasts interactive capabilities, enabling users to communicate their queries on surface changes.

Change-Agent can adeptly interpret changes according to user needs, intelligently analyze and process the change interpre- tation information, and ultimately deliver tailored outcomes aligning with user expectations. The interactive Change-Agent bridges the gap between users and remote sensing expertise.

As illustrated in Fig.1 (b), the MCI model is a pivotal cornerstone toward the realization of the Change-Agent. It enables the Change-Agent to perceive visual changes com- prehensively, serving as the change perception foundation of the Change-Agent. However, there has been no prior work investigating the multi-task learning of change detection and change captioning. To address this gap, we built the LEVIR- MCI dataset. Each pair of bi-temporal remote sensing images in this dataset contains both pixel-level change detection masks and semantic-level change descriptions. Building upon this dataset, we propose a novel MCI model with two branches of change detection and change captioning. Within the two branches, we propose BI-temporal Iterative Interaction (BI3) layers utilizing Local Perception Enhancement (LPE) and the Global Difference Fusion Attention (GDFA) modules to enhance the model’s discriminative feature representation capabilities. The trained MCI model equips our Change-Agent with eyes, enabling the agent to realize comprehensive change interpretations of surface changes.

Recent Advancements In Llms Such As Chatgpt And

Llama2 have been remarkable. Trained on vast corpora, LLMs have acquired extensive knowledge and demonstrated powerful capabilities in instruction comprehension, planning, reasoning, question answering, and text generation , .

These notable progress have provided robust support for our LLM-based Change-Agent. In this work, the LLM serves as the brain of the Change-Agent, assuming a pivotal role in internal scheduling, planning, and user-intent understanding.

Integration with LLMs imbues our agent with enhanced flex- ibility and intelligence, enabling a nuanced understanding of user intentions and provision of customized change interpre- tation and intelligent analysis services such as changed object counting, estimation of causes, prediction of changes, etc.

Our Change-Agent facilitates accurate and comprehensive change interpretation of RS images, offering novel perspec- tives for analyzing changes on the Earth’s surface. It reduces the workload and time costs for researchers while enhancing the efficiency and convenience of interaction between users and remote sensing data. Extensive experiments demonstrate the effectiveness of the proposed MCI model and highlight the promising application value of our Change-Agent.

Our Contributions Can Be Summarized As Follows:

• We develop an interactive Change-Agent, which uses external tools to achieve comprehensive interpretation and analysis of surface changes according to the user’s instructions. It has intelligent dialogue and customized service capabilities, opening a new avenue for intelligent remote-sensing applications.

• We propose a multi-task learning-based MCI model with dual branches for change detection and change cap- tioning. This model can simultaneously provide pixel- level and semantic-level change interpretation informa- tion, serving as the perception tool of the Change-Agent.

• We built an MCI dataset named LEVIR-MCI, which con- tains bi-temporal images, corresponding change detection masks and descriptive sentences. The dataset provides a crucial data foundation for exploring multi-task learning for change detection and change captioning.

Ii. Related Work

A comprehensive change interpretation model should pro- vide both pixel-level and semantic-level change information. However, previous work has primarily focused on either change detection or change captioning alone. Change detec- tion aims to generate pixel-level change maps revealing the locations of changes in bi-temporal images, while change captioning aims to describe the changes via language.

Traditionally

categorized into algebra-based, transformation-based, and classification-based methods. These methods relied on tech-

Niques Such As Change Vector Analysis (Cva) , ,

principal component analysis (PCA) , , multivariate

3

alteration detection (MAD) , , and post-classification

Comparisons , . Traditional Methods Have Laid The

foundation for modern change detection techniques . With the emergence of deep learning –, the field of remote sensing change detection has witnessed significant ad- vancements –. Compared to traditional methods, deep learning significantly improves the performance of change detection with its superior multi-temporal feature learning capability. These methods utilize various architectures includ-

–, And Transformers , –. For Instance, Chen

et al. introduced a bi-temporal image Transformer (BIT) , where images are represented as semantic tokens, and the Transformer encoder and decoder refine visual tokens to identify changed areas efficiently and effectively. Li et al.

proposed an end-to-end TransUNetCD model based on Trans- former and UNet structure. The model utilizes a Transformer encoder to extract global contextual features. A CNN-based decoder upsamples the multi-scale features to generate the change map. Bandara et al. proposed a pure transformer model named ChangeFormer , whose backbone is a siamese Seg- Former . Zhang et al. propose a deep siamese network with multi-scale multi-attention. They enhance global repre- sentation using a contextual attention module and fuse multi- scale features from the siamese feature extractor for detecting objects of varying sizes and irregularities. Compared to binary change detection, semantic change detection considers the semantic information of change categories, which is presented in the form of semantic masks. Some works, such as , , have significantly advanced the development of semantic change detection. However, higher-level semantic information, such as the relationships between changed objects, their colors, and spatial distribution, remains challenging to directly obtain.

Some surveys, such as , provide a more comprehensive review of change detection in deep learning.

B. Remote Sensing Change Captioning

Remote sensing change captioning (RSCC) is a recently emerged multimodal task involving remote sensing image pro- cessing and natural language generation. Given the intricacy of this task, prevailing methodologies predominantly leverage deep learning techniques. Early methods in this domain largely drew inspiration from the encoder-decoder framework com- monly utilized in the image captioning task –. Within this framework, the encoder is responsible for extracting fea- tures from bi-temporal images and performing the difference- aware fusion of these features. The decoder typically employs recurrent neural networks or Transformers to convert visual features into natural language. Compared to tasks like image captioning, which extract semantic representations from single images, change description faces the significant challenge of extracting robust semantic change representations.

Chouaf and Hoxha et al. , pioneered remote sensing change captioning by introducing a model that employs a pre- trained CNNs as the encoder and tried to utilize an RNN and Support Vector Machines (SVMs) as the decoder to generate

6665 9645

Fig. 2. The comparison of the proposed LEVIR-MCI dataset and the previous LEVIR-CC dataset. The LEVIR-MCI dataset is an extension of LEVIR-CC. textual descriptions. To improve the change-awareness of the model, they proposed an image-based fusion strategy and a feature-based fusion strategy. To facilitate research on change captioning, Liu et al. introduced a large LEVIR-CC dataset and benchmarked some methods from other fields , . Besides, they proposed a Transformer-based method named RSICCformer. The approach incorporates Siamese cross-encoding modules and multistage bitemporal fusion modules, emphasizing change regions through differencing features and capturing multiple changes of interest.

In a further advancement, some attention-based improve-

Ments Have Emerged , –. For Instance, Psnet

enhances the perception of changed objects of varying sizes using a progressive scale-aware network with difference perception layers and scale-aware reinforcement modules.

Chang et al. proposed an attention network containing a hierarchical self-attention module followed by a residual unit for identifying change-related attributes and constructing the semantic change embedding. Unlike the encoder-decoder framework, Liu et al. proposed a decoupling paradigm by designing an image-level classifier and a feature-level encoder. Besides, a multi-prompt learning strategy is proposed to effectively exploit an LLM for captioning.

Iii. Levir-Mci Dataset

To cultivate a robust MCI model, foundational to the Change-Agent, we propose an MCI dataset named LEVIR- MCI (LEVIR Multi-level Change Interpretation) 1, featuring both pixel-level change information in the form of change detection masks and semantic-level insights encapsulated in descriptive sentences. The LEVIR-MCI dataset contains 10077 bi-temporal images. Each image has a spatial size of 256 × 256 pixels with a high resolution of 0.5 m/pixel and has a corresponding annotated mask and five annotated captions.

The dataset will be publicly available at: https://github.com/

Chen-Yang-Liu/Change-Agent

The proposed LEVIR-MCI dataset is an extension of our previously established change captioning dataset, LEVIR-CC . As shown in Fig. 2, we further provided each pair of bi-temporal images with additional change detection masks highlighting changed roads and changed buildings. Unlike its 1LEVIR is our laboratory name: the Learning, Vision, and Remote Sensing Laboratory.

4

Caption: A lake has formed in the upper right corner of the sparse jungle. Caption: The sparse vegetation is replaced by some roads and large constructions with parking lots.

Caption: Trees and plants are removed and a turning road is built with villas built along on both sides.

Caption: Two Parallel Roads Have Been Constructed

beside the original street near the forest. Caption: a residential area with many houses and roads appears.

Caption: Several roads appear and two buildings appear and a swimming pool appears. Fig. 3.

Examples of the LEVIR-MCI dataset. Each pair of bi-temporal images is provided with a change detection mask and one of the five sentences describing changes. In the change detection mask, changed buildings are highlighted in red, while changed roads are depicted in yellow.

Color In Bins

Fig. 4. Distribution of the scale and deformation of changed roads and buildings. The “area” on the horizontal axis represents the area of a single object, and the “bbox area” on the vertical axis represents the area of the corresponding rectangular bounding box. The value of the color bins represents the number of object instances. The dispersion of points offers insights into the diversity of object scale and deformation.

Table I

The Number of changed roads and buildings in the LEVIR-MCI dataset. The dataset comprises over 40,000 annotated instances of changed roads and buildings.

8.92

predecessor, the LEVIR-MCI dataset provides diverse annota- tions from different interpretation perspectives for each pair of images, further enhancing its utility for comprehensive change interpretation.

Fig. 3 presents some examples from our proposed LEVIR- MCI. Each pair of bi-temporal images within the dataset is annotated by a change detection mask delineating pixel- level alterations, along with one of five descriptive sentences elucidating the semantic-level changes. Such complementary annotations provide a multifaceted view of change interpre- tation, thereby bolstering the comprehensiveness of change interpretation models. To facilitate a nuanced understanding of the dataset, and considering our previous research has analysed the change captions, we will elucidate additional insights into the annotated change detection masks.

A. Number Of Changed Objects

In Table I, we meticulously analyse changed object masks in the LEVIR-MCI dataset. The dataset comprises over 40,000 annotated instances of changed roads and buildings. While the number of changed roads is fewer than buildings, generally exhibit longer spans and cover larger areas, as evidenced in Fig. 4. With such a vast collection of annotated masks, our LEVIR-MCI dataset can also serve as a fertile data ground for developing innovative multi-category change detection methodologies.

B. Scale And Deformation Of Changed Objects

An insightful examination of object scale and deformation is conducted through an analysis of road and building areas and the corresponding rectangular bounding boxes, as illustrated in Fig. 4. The “area” on the horizontal axis represents the

Some Houses Are Neatly Built On Both

sides of two roads.

Both Sides Of Two

roads.

(B)

Fig. 5. The overview of Change-Agent is shown in (a). The Change-Agent is equipped with an MCI model and an LLM, serving as its eyes and brain, respectively. The proposed LEVIR-MCI dataset provides a data foundation for training the MCI model. (b) shows the overall structure of the MCI model.

area of a single object, and the “bbox area” on the vertical axis represents the area of the corresponding bounding box. The dispersion of points offers insights into the diversity of object scale and deformation. Notably, points of roads exhibit a relatively broader spectrum with dispersed diversity, attributable to their narrow and curved nature. In contrast, points of buildings display a more concentrated distribution, reflecting their predominantly rectangular form. This anal- ysis elucidates the intricate interplay between object scale and shape, shedding light on the nuanced characteristics of changed objects.

Iv. Methodology

In Fig. 5, we present the overview of the proposed Change- Agent. We utilize the MCI model as eyes and an LLM as the brain to build our Change-Agent. The MCI model enables the Change-Agent to perceive visual changes compre- hensively, serving as the change perception foundation of the Change-Agent. The constructed LEVIR-MCI dataset supports the multi-task training of the MCI model, which contains a change detection branch and a change captioning branch.

As another pivotal role of the agent, the LLM can use its inherent rich knowledge to implement agent scheduling and provide insightful analyses and decision-making support. In the subsequent section, we will introduce the composition of the MCI model and the LLM-based scheduling of the agent.

A. Multi-Level Change Interpretation Model

The MCI model serves as a central component of the Change-Agent, tasked with extracting and interpreting change information from bi-temporal remote sensing images. The proposed LEVIR-MCI dataset lays the groundwork for train- ing the MCI model. The MCI model adopts a dual-branch architecture with a shared bottom, focusing on two pivotal tasks: change detection and change captioning. Specifically, a Siamese backbone network extracts multi-scale visual features from bi-temporal images, enabling learning in two branches.

The lower-level features offer detailed information, while higher-level features are rich in semantics. Our change detec- tion branch utilizes multi-scale features to refine change mask predictions, while the change captioning branch leverages the highest-level visual features to generate descriptive sentences.

Through multi-task learning, we train a robust MCI model capable of simultaneously generating change detection masks and change captions.

B. Bi-Temporal Iterative Interaction Layer

In the change detection and change captioning branches, we propose a novel BI-temporal Iterative Interaction (BI3) layer to effectively enhance and fuse bi-temporal features. The structure of the BI3 layer is illustrated in Fig. 6. The BI3 layer utilizes the Local Perception Enhancement (LPE) module and the Global Difference Fusion Attention (GDFA) module to extract the discriminative features of interest.

Motivated by the analysis presented in Section III-B regard- ing the various scales and deformations of roads and buildings, we propose the LPE module with a residual structure. The LPE module employs convolution kernels of different sizes to ex- tract multiple feature maps across different scales. This design enriches the diversity of feature information and improves the model’s local feature perception capabilities. Assuming the bi-

Gdfa

Fig. 6. The structure of the Bi-temporal Iterative Interaction (BI3) layer. The BI3 layer utilizes the Local Perception Enhancement (LPE) and the Global Difference Fusion Attention (GDFA) to enhance the features of interest.

temporal features are denoted as XT1 ∈RN×C and XT2 ∈RN×C, the LPE module ΦLPE(·) can be formulated as follows:

(5)

where conv1×1, conv3×3, conv5×1, conv1×5 represent 1 × 1, 3 × 3, 5 × 1, and 1 × 5 convolution kernels. concat stands for concatenation operation, BN (Batch Normalization) for normalization, and ReLU (Rectified Linear Unit) for nonlinear activation function.

The GDFA module leverages differencing features to gen- erate spatial attention weights and perform the interaction and fusion between features. This facilitates the model to focus on changes of interest and ignore irrelevant disturbances. Taking the left GDFA module in Fig. 6 as an example, assuming Xlp1 ∈RN×C and Xlp2 ∈RN×C are the bi-temporal features obtained from the LPE module, the GDFA module ΦGDFA(·)

(11)

where σ is the softmax function, Wq, Wk, Wv, Wd ∈RC×d are trainable weight matrices. bq, bk, bv, bd ∈Rd are trainable bias.

�

...

Mask

Fig. 7. The structure of the change detection branch. “DeConv” denotes the deconvolution. The multi-scale features are gradually fused from bottom to top for refined mask prediction.

Bi3 Layer

...

K,V

Fig. 8. The structure of the change captioning branch. The projection layer further bridges the conversion from the visual domain to the textual domain. The Transformer decoder generates captions.

C denotes the dimension of Xlpi (i=1,2). d is the dimension of transformed features. Through the combination of LPE module and GDFA mod- ule, the BI3 layer improves the model’s feature representation and change discrimination capabilities. Following the LPE and GDFA modules, Layer Normalization (LN) is applied to the bi-temporal features. Subsequently, a Multi-Layer Per- ceptron (MLP) with residuals further refines the normalized features to obtain enhanced bi-temporal features.

C. Change Detection And Captioning Branch

The structure of the change detection branch is depicted in Fig. 7. Leveraging multi-scale bi-temporal features extracted from the backbone network, this branch facilitates refined mask prediction. Specifically, multiple Bi-temporal Iterative Interaction (BI3) layers with residual connections iteratively enhance and refine bi-temporal high-level features, effectively capturing semantic changes. Since the low-level features con- tain more detailed information, they are crucial for refined change boundary detection. We further incorporate multiple Convolution-Based Bi-temporal Fusion (CBF) modules for bi-temporal feature fusion across four scales. Subsequently, through deconvolution, features are progressively integrated from bottom to top, bolstering the model’s discrimination ca- pability for changes and enhancing change detection accuracy.

Cc Branch

......

Change Description

......

Llm

Fig. 9. The LLM plans how to complete the task according to the user’s instructions. We provide a suite of Python tools. The LLM drafts Python programs, subsequently executed by a Python interpreter to accomplish tasks beyond its inherent capabilities.

(14)

where cos(·) denotes cosine similarity operation. The change captioning branch focuses on semantic-level change interpretation, translating visual changes into textual descriptions, as depicted in Fig. 8. This branch leverages multiple BI3 layers to interactively process high-level semantic features extracted from the backbone network, thereby obtain- ing bi-temporal visual features revealing changes of interest.

Subsequently, a convolution-based projection layer further processes the bi-temporal features to facilitate the transition from the visual domain to the textual domain. Finally, these processed features are fed into a Transformer decoder to generate descriptive sentences elucidating the changes. The processing flow of the projection layer ΦPr(·) can be expressed

D. Llm: Brain Of Change-Agent

Recently, LLMs have garnered significant attention for their remarkable achievements. By harnessing extensive web knowledge, LLMs have demonstrated substantial promise in attaining human-level intelligence, exhibiting noteworthy ca- pabilities in tasks encompassing instruction comprehension, planning, reasoning, and proficient natural language inter-

Actions With Humans , –. Building Upon These

advancements, many studies are beginning to utilize LLMs as central controllers for building autonomous agents –. Inspired by these developments, we harness the capabilities of LLMs as the cognitive core responsible for orchestrating the scheduling of our Change-Agent. As depicted in Fig. 9, the LLM meticulously plans task execution based on user instructions. Despite excelling at text-related tasks, LLMs lack inherent visual perception capabilities. To bridge this gap and enable change interpretation and analysis akin to human capability, we equip our agent with a suite of Python tools. These tools include a visual feature extraction back- bone, change detection branch, change captioning branch, and relevant Python libraries. Leveraging these tools, the LLM autonomously crafts Python programs, subsequently executed by a Python interpreter to automatically accomplish tasks beyond its intrinsic capabilities without the need for human intervention. Subsequently, the LLM processes the results and furnishes feedback to the user. In addition to furnishing change masks and descriptions, our agent can implement object counting, image processing, change cause estimation, prediction of future changes, and more. In short, the Change- Agent adeptly provides users with pertinent information by leveraging the rich knowledge of the LLM and additional tools.

To facilitate the accurate generation of formatted Python code and tool invocation by the LLM, a meticulously crafted text prompt is essential. Inspired by the prompt utilized in , we devised a text prompt instructing the model on proper tool utilization. This prompt is input to the LLM in the role of a system instruction, as shown in Fig. 10.

A. Objective Function For Multi-Task Training

During the training phase of our MIC model, we perform multi-task learning on both change detection and change captioning. Utilizing the meticulously constructed LEVIR- MCI dataset, we iteratively update the model parameters in a supervised learning framework.

For the change detection, we utilize the cross-entropy loss function to quantify the dissimilarity between predicted change masks and ground truth annotations. Specifically, the loss function can be mathematically defined as follows:

(18)

where C is the number of classes. H and W are the height

Cls ] Is The

predicted probability vector at the pixel position (h,w). eyhw

Hw ] Is The One-Hot Vector Representation Of The

ground-truth at the pixel position (h,w). Similarly, for change captioning, we utilize the cross- entropy function to compute the loss between predicted sen- tences and their corresponding ground truth sentences. The formulation of the captioning loss function can be expressed

(19)

where L represents the total number of word tokens, V

8

You are a assistant who can utilize external tools. {'Visual_Change_Process_PythonInterpreter': """This action tool is used to detection changed object masks or describle between two images! It also is a Python executor capable of running Python scripts.

The input for this action tool must be valid Python function. The function name must be 'solution'. And the function code corresponds to your thinking process.

Just write the function 'def solution()'. For the input for this action tool, example format is

# Step 1

mid_variable = func(variable_names_with_real_meaning)

Mid_Variable = Func(Mid_Variable)

# step .....

Note:

In addition to the commonly used python dependency packages, you can use a custom library 'Change_Perception' from tools import Change_Perception(), which contains the

Following Function:

1. **`change_detection(path_A, path_B, savepath_mask)`**:

- **Parameters**:

- `path_A`: Path to the first image. - `path_B`: Path to the second image. - `savepath_mask`: Path to save the mask image.

- **Returns**:

- A mask map representing the changed areas. The mask is a numpy array with dimensions (256,256), where each pixel value represents the following:

- 1 Stands For Changed Road, And

- 2 stands for changed building.

- **Parameters**:

- `path_A`: Path to the first image. - `path_B`: Path to the second image.

- **Returns**:

- A String: Sentences describing the changes between the two images.

- **Parameters**:

- `change_mask`: The mask from the `change_detection` function. - `object`: The object type to be counted. It can be either 'building' or 'road'.

- **Returns**:

- The number of changed objects. Here is a sample how use the above function to detect changed buildings and save the

```Python

mask = Change_Perception().change_detection(path_A, path_B, savepath_mask) mask_rgb = np.zeros((mask.shape, mask.shape, 3), dtype=np.uint8) mask_rgb[mask == 2] = [0, 0, 255] # '2' stands for changed building (red)

Cv2.Imwrite(Savepath_Mask, Mask_Rgb)

```"""}.

```

Thought: Think about what you need to solve. Do you need to use any tools? Action: Provide the name of the tool you want to use. It should be one of:

[[ 'Visual_Change_Process_Pythoninterpreter']]

Action Input: Specify the input required for the action tool.

```

The response after utilizing tools should using the following format:

```

Response: Display the results obtained after calling the tool.

``

If you already know the answer, or if you don't need to use any tools, please reply using

```

Thought: Explain your thought process to arrive at the final answer.

Begin!

Fig. 10. The prompt input to LLM in the role of a system instruction. indicating whether l-th word belongs to vocabulary index v,

L

denotes the predicted probability of l-th word being assigned to vocabulary index v. To balance the losses of these two tasks during model training, we adopt a normalization approach, scaling the losses of both tasks to the same order of magnitude. This ensures that each task contributes equally to the overall loss function and facilitates effective simultaneous optimization of the change detection and change captioning. Mathematically, the total loss

(20)

where detach(·) denotes the operation to detach the gradient flow of the loss.

B. Evaluation Metrics

To evaluate the performance of our MCI model in detecting multi-category changes in remote sensing images, we use the Mean Intersection over Union (MIoU). The MIoU metric as- sesses the spatial overlap between predicted change masks and ground-truth masks, providing a reliable measure of pixel-level change detection accuracy. Mathematically, the calculation of

(22)

Where N is the total number of classes (i.e., Background, road, and building). For the i-th class, TPi, FPi, and FNi represent the number of true positive, false positive, and false negative, respectively.

To evaluate the performance of our MCI model in describing changes between bi-temporal images, we adopted several eval- uation metrics commonly used in previous change captioning

• Bleu-N : The Bleu-N Metric Measures The N-Gram

similarity between generated sentences and ground-truth sentences. We adopt n=1, 2, 3, and 4 to compute the BLEU-n score.

• Meteor : The Meteor Computes The Harmonic

mean of precision and recall of single words. Besides, it incorporates a penalty factor to consider the fluency of generated sentences.

• Rougel : The Rougel Metric Measures The Sim-

ilarity of the longest common subsequence between gen- erated sentences and ground-truth sentences. It is more concerned with the recall rate and is suitable for evaluat- ing long sentences.

• Cider-D : The Cider Metric Treats Each Sentence

as a document and represents it in the form of Term Fre- quency Inverse Document Frequency (TF-IDF) vectors. After computing TF-IDF for each n-gram, CIDEr-D is obtained by performing the cosine similarity.

By employing this comprehensive suite of evaluation met- rics, we aim to provide a thorough assessment of the change

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.