Complete Technical Paper Content — Ai Meeting Summarization
1. Executive Abstract, Problem Formulation & Background
Smart Meeting Summarization and Action Item Detection Smart Meeting Summarization and Action Item Detection is an AI-based system that converts meeting recordings into concise summaries and extracts tasks, deadlines, and assignees. Using Whisper, BART, PEGASUS, and BERT, it improves productivity by automating meeting analysis and prioritizing action items efficiently. 📖 Overview Smart Meeting Summarization and Action Item Detection is an AI-powered system designed to automate meeting analysis by generating concise summaries and extracting important action items from virtual and hybrid meetings. The system converts meeting recordings into structured textual insights using advanced Artificial Intelligence, Natural Language Processing (NLP), Speech Recognition, and Deep Learning techniques. It integrates OpenAI Whisper for speech transcription, BART and PEGASUS for abstractive summarization, and BERT for entity extraction and action-item detection. The project aims to reduce manual note-taking efforts, improve productivity, enhance collaboration, and support faster decision-making in professional, educational, and industrial environments. 🚀 Features Automatic Speech Recognition using Whisper Meeting transcription from audio/video recordings Abstractive meeting summarization Action item extraction Deadline and assignee detection Task prioritization system Multi-speaker meeting handling Long transcript summarization Interactive web interface AI-powered meeting analytics ❗ Problem Statement Traditional meeting documentation methods are time-consuming and often fail to capture important discussions, decisions, and tasks accurately. Existing summarization systems struggle with multi-speaker conversations, overlapping speech, noisy audio, and long unstructured discussions. This project addresses these challenges by developing an intelligent automated system capable of generating accurate summaries and extracting actionable tasks with priorities and deadlines. 🎯 Objectives Convert meeting audio into accurate text transcripts Generate concise abstractive summaries Extract action items automatically Detect deadlines and assignees Prioritize tasks based on urgency Reduce manual meeting documentation effort Improve productivity and meeting management 🏗 System Architecture The system architecture consists of multiple processing layers: Audio/Video Input Layer Audio Extraction Layer Speech Recognition Layer NLP Summarization Layer Action Item Detection Layer Entity Extraction Layer Task Prioritization Layer Streamlit Web Interface The architecture integrates Whisper ASR, BART, PEGASUS, and BERT models to generate structured meeting insights. ⚙️ Methodology The proposed system operates through a multi-stage AI pipeline: Input meeting audio/video recordings Extract audio using MoviePy Convert speech into text using Whisper ASR Generate summaries using BART and PEGASUS Evaluate summary quality Extract action items using BERT and rule-based techniques Detect entities such as action, assignee, and deadlines Prioritize tasks using scoring mechanisms Generate final structured output 🧠 Algorithm Step 1: Input Video Processing Load MP4 meeting recordings Extract audio using MoviePy Convert audio into MP3 format Step 2: Speech Recognition Process audio using Whisper Large-v2 Generate meeting transcripts Step 3: Meeting Summarization Pass transcript into BART and PEGASUS Compare summary quality Select best-performing summary Step 4: Action Item Detection Apply rule-based keyword detection Extract ML features Identify actionable tasks Step 5: Entity Extraction Use BERT with BIO tagging Detect: Action Assignee Deadline Step 6: Task Prioritization Assign urgency scores Rank tasks based on contextual importance Step 7: Final Output Generate summaries Generate action-item list Generate priority-based tasks 📂 Dataset Description AMI Meeting Corpus The project uses the AMI Meeting Corpus dataset containing approximately 140 multi-speaker meetings recorded in English. Dataset Includes Audio recordings Video recordings Meeting transcripts XML annotations Speaker interactions Supported Formats MP4 WAV XML RTF The dataset is suitable for meeting summarization, speech recognition, and action-item extraction research. 🛠 Technologies Used Programming Language Python Libraries & Frameworks Transformers Hugging Face PyTorch MoviePy NumPy Pandas NLTK spaCy Streamlit 🤖 Models Used Model Purpose Whisper Speech Recognition BART Abstractive Summarization PEGASUS Long Text Summarization BERT Action Item Detection 🔍 Whisper OpenAI Whisper is used for Automatic Speech Recognition (ASR). It converts meeting audio into text transcripts while handling noisy environments and multi-speaker conversations. 📝 BART BART is a transformer-based abstractive summarization model that generates concise summaries from long conversational transcripts. 📄 PEGASUS PEGASUS is designed specifically for text summarization tasks and produces highly coherent and context-aware summaries. 🧾 BERT BERT with BIO tagging is used for entity extraction and action-item detection. It identifies tasks, deadlines, and assigned persons. 💻 Installation Clone Repository git clone https://github.com/Riyashelke/smart-meeting-summarization.git Move into Project Directory cd smart-meeting-summarization Create Virtual Environment python -m venv venv ▶️ Activate Environment macOS/Linux source venv/bin/activate Windows venv \S cripts \a ctivate 📦 Install Dependencies pip install -r requirements.txt 🚀 Usage Run Main Pipeline python pipeline_combined.py Run API python pipeline_api.py 📊 Expected Output The system generates: Meeting transcript Abstractive summary Action items Assignee information Deadlines Priority-ranked task list 📈 Results and Analysis The project compares the performance of BART and PEGASUS models based on: Processing time Summary quality Semantic coherence Information retention Context understanding Observations PEGASUS achieved better summary quality (~0.84) BART achieved summary quality (~0.78) PEGASUS demonstrated more stable processing performance The hybrid action-item detection system achieved over 90% accuracy in tested meeting cases 🌍 Applications Corporate meeting analysis Online collaboration platforms Educational discussions Industrial workflow management Productivity enhancement systems AI-powered documentation tools ✅ Advantages Reduces manual note-taking Saves time Improves productivity Generates structured insights Supports long meeting analysis Handles multi-speaker conversations Automates task extraction 🔮 Future Enhancements Real-time meeting summarization Multilingual support Cloud deployment Advanced speaker diarization Live meeting assistant integration Sentiment analysis dashboard Mobile application support 📌 Conclusion The Smart Meeting Summarization and Action Item Detection system successfully automates the process of meeting transcription, summarization, and task extraction using AI and NLP technologies. By integrating Whisper, BART, PEGASUS, and BERT models, the system generates accurate summaries, extracts actionable insights, and prioritizes tasks effectively. The project improves productivity, reduces manual effort, and provides a scalable solution for modern meeting management. 👨💻 Authors Riya Shelke Vaishnavi Ranaware Namoha Goyal Dr. Pranali Kosamkar (Guide) MIT World Peace University 📚 References AMI Meeting Corpus OpenAI Whisper Hugging Face Transformers BART Research Paper PEGASUS Research Paper BERT Research Paper QMSum Dataset SAMSum Dataset 📄 License This project is developed for academic and research purposes. 🔗 GitHub Repository Repository Link: https://github.com/Riyashelke/smart-meeting-summarization
Smart Meeting Summarization and Action Item Detection is an AI-based system that converts meeting recordings into concise summaries and extracts tasks, deadlines, and assignees. Using Whisper, BART, PEGASUS, and BERT, it improves productivity by automating meeting analysis and prioritizing action items efficiently.
Smart Meeting Summarization and Action Item Detection is an AI-powered system designed to automate meeting analysis by generating concise summaries and extracting important action items from virtual and hybrid meetings. The system converts meeting recordings into structured textual insights using advanced Artificial Intelligence, Natural Language Processing (NLP), Speech Recognition, and Deep Learning techniques. It integrates OpenAI Whisper for speech transcription, BART and PEGASUS for abstractive summarization, and BERT for entity extraction and action-item detection.
The project aims to reduce manual note-taking efforts, improve productivity, enhance collaboration, and support faster decision-making in professional, educational, and industrial environments.
Traditional meeting documentation methods are time-consuming and often fail to capture important discussions, decisions, and tasks accurately. Existing summarization systems struggle with multi-speaker conversations, overlapping speech, noisy audio, and long unstructured discussions. This project addresses these challenges by developing an intelligent automated system capable of generating accurate summaries and extracting actionable tasks with priorities and deadlines.
The architecture integrates Whisper ASR, BART, PEGASUS, and BERT models to generate structured meeting insights.
The project uses the AMI Meeting Corpus dataset containing approximately 140 multi-speaker meetings recorded in English.
The dataset is suitable for meeting summarization, speech recognition, and action-item extraction research.
OpenAI Whisper is used for Automatic Speech Recognition (ASR). It converts meeting audio into text transcripts while handling noisy environments and multi-speaker conversations.
2. Related Work, Literature Survey & Algorithmic Foundations
BART is a transformer-based abstractive summarization model that generates concise summaries from long conversational transcripts.
PEGASUS is designed specifically for text summarization tasks and produces highly coherent and context-aware summaries.
BERT with BIO tagging is used for entity extraction and action-item detection. It identifies tasks, deadlines, and assigned persons.
The Smart Meeting Summarization and Action Item Detection system successfully automates the process of meeting transcription, summarization, and task extraction using AI and NLP technologies. By integrating Whisper, BART, PEGASUS, and BERT models, the system generates accurate summaries, extracts actionable insights, and prioritizes tasks effectively. The project improves productivity, reduces manual effort, and provides a scalable solution for modern meeting management.
Smart Meeting Summarization and Action Item Detection is an AI-based system that converts meeting recordings into concise summaries and extracts tasks, deadlines, and assignees. Using Whisper, BART, PEGASUS, and BERT, it improves productivity by automating meeting analysis and prioritizing action items efficiently.
Smart Meeting Summarization and Action Item Detection Smart Meeting Summarization and Action Item Detection is an AI-based system that converts meeting recordings into concise summaries and extracts tasks, deadlines, and assignees. Using Whisper, BART, PEGASUS, and BERT, it improves productivity by automating meeting analysis and prioritizing action items efficiently. 📖 Overview Smart Meeting Summarization and Action Item Detection is an AI-powered system designed to automate meeting analysis by generating concise summaries and extracting important action items from virtual and hybrid meetings. The system converts meeting recordings into structured textual insights using advanced Artificial Intelligence, Natural Language Processing (NLP), Speech Recognition, and Deep Learning techniques. It integrates OpenAI Whisper for speech transcription, BART and PEGASUS for abstractive summarization, and BERT for entity extraction and action-item detection. The project aims to reduce manual note-taking efforts, improve productivity, enhance collaboration, and support faster decision-making in professional, educational, and industrial environments. 🚀 Features Automatic Speech Recognition using Whisper Meeting transcription from audio/video recordings Abstractive meeting summarization Action item extraction Deadline and assignee detection Task prioritization system Multi-speaker meeting handling Long transcript summarization Interactive web interface AI-powered meeting analytics ❗ Problem Statement Traditional meeting documentation methods are time-consuming and often fail to capture important discussions, decisions, and tasks accurately. Existing summarization systems struggle with multi-speaker conversations, overlapping speech, noisy audio, and long unstructured discussions. This project addresses these challenges by developing an intelligent automated system capable of generating accurate summaries and extracting actionable tasks with priorities and deadlines. 🎯 Objectives Convert meeting audio into accurate text transcripts Generate concise abstractive summaries Extract action items automatically Detect deadlines and assignees Prioritize tasks based on urgency Reduce manual meeting documentation effort Improve productivity and meeting management 🏗 System Architecture The system architecture consists of multiple processing layers: Audio/Video Input Layer Audio Extraction Layer Speech Recognition Layer NLP Summarization Layer Action Item Detection Layer Entity Extraction Layer Task Prioritization Layer Streamlit Web Interface The architecture integrates Whisper ASR, BART, PEGASUS, and BERT models to generate structured meeting insights. ⚙️ Methodology The proposed system operates through a multi-stage AI pipeline: Input meeting audio/video recordings Extract audio using MoviePy Convert speech into text using Whisper ASR Generate summaries using BART and PEGASUS Evaluate summary quality Extract action items using BERT and rule-based techniques Detect entities such as action, assignee, and deadlines Prioritize tasks using scoring mechanisms Generate final structured output 🧠 Algorithm Step 1: Input Video Processing Load MP4 meeting recordings Extract audio using MoviePy Convert audio into MP3 format Step 2: Speech Recognition Process audio using Whisper Large-v2 Generate meeting transcripts Step 3: Meeting Summarization Pass transcript into BART and PEGASUS Compare summary quality Select best-performing summary Step 4: Action Item Detection Apply rule-based keyword detection Extract ML features Identify actionable tasks Step 5: Entity Extraction Use BERT with BIO tagging Detect: Action Assignee Deadline Step 6: Task Prioritization Assign urgency scores Rank tasks based on contextual importance Step 7: Final Output Generate summaries Generate action-item list Generate priority-based tasks 📂 Dataset Description AMI Meeting Corpus The project uses the AMI Meeting Corpus dataset containing approximately 140 multi-speaker meetings recorded in English. Dataset Includes Audio recordings Video recordings Meeting transcripts XML annotations Speaker interactions Supported Formats MP4 WAV XML RTF The dataset is suitable for meeting summarization, speech recognition, and action-item extraction research. 🛠 Technologies Used Programming Language Python Libraries & Frameworks Transformers Hugging Face PyTorch MoviePy NumPy Pandas NLTK spaCy Streamlit 🤖 Models Used Model Purpose Whisper Speech Recognition BART Abstractive Summarization PEGASUS Long Text Summarization BERT Action Item Detection 🔍 Whisper OpenAI Whisper is used for Automatic Speech Recognition (ASR). It converts meeting audio into text transcripts while handling noisy environments and multi-speaker conversations. 📝 BART BART is a transformer-based abstractive summarization model that generates concise summaries from long conversational transcripts. 📄 PEGASUS PEGASUS is designed specifically for text summarization tasks and produces highly coherent and context-aware summaries. 🧾 BERT BERT with BIO tagging is used for entity extraction and action-item detection. It identifies tasks, deadlines, and assigned persons. 💻 Installation Clone Repository git clone https://github.com/Riyashelke/smart-meeting-summarization.git Move into Project Directory cd smart-meeting-summarization Create Virtual Environment python -m venv venv ▶️ Activate Environment macOS/Linux source venv/bin/activate Windows venv \S cripts \a ctivate 📦 Install Dependencies pip install -r requirements.txt 🚀 Usage Run Main Pipeline python pipeline_combined.py Run API python pipeline_api.py 📊 Expected Output The system generates: Meeting transcript Abstractive summary Action items Assignee information Deadlines Priority-ranked task list 📈 Results and Analysis The project compares the performance of BART and PEGASUS models based on: Processing time Summary quality Semantic coherence Information retention Context understanding Observations PEGASUS achieved better summary quality (~0.84) BART achieved summary quality (~0.78) PEGASUS demonstrated more stable processing performance The hybrid action-item detection system achieved over 90% accuracy in tested meeting cases 🌍 Applications Corporate meeting analysis Online collaboration platforms Educational discussions Industrial workflow management Productivity enhancement systems AI-powered documentation tools ✅ Advantages Reduces manual note-taking Saves time Improves productivity Generates structured insights Supports long meeting analysis Handles multi-speaker conversations Automates task extraction 🔮 Future Enhancements Real-time meeting summarization Multilingual support Cloud deployment Advanced speaker diarization Live meeting assistant integration Sentiment analysis dashboard Mobile application support 📌 Conclusion The Smart Meeting Summarization and Action Item Detection system successfully automates the process of meeting transcription, summarization, and task extraction using AI and NLP technologies. By integrating Whisper, BART, PEGASUS, and BERT models, the system generates accurate summaries, extracts actionable insights, and prioritizes tasks effectively. The project improves productivity, reduces manual effort, and provides a scalable solution for modern meeting management. 👨💻 Authors Riya Shelke Vaishnavi Ranaware Namoha Goyal Dr. Pranali Kosamkar (Guide) MIT World Peace University 📚 References AMI Meeting Corpus OpenAI Whisper Hugging Face Transformers BART Research Paper PEGASUS Research Paper BERT Research Paper QMSum Dataset SAMSum Dataset 📄 License This project is developed for academic and research purposes. 🔗 GitHub Repository Repository Link: https://github.com/Riyashelke/smart-meeting-summarization
Smart Meeting Summarization and Action Item Detection is an AI-based system that converts meeting recordings into concise summaries and extracts tasks, deadlines, and assignees. Using Whisper, BART, PEGASUS, and BERT, it improves productivity by automating meeting analysis and prioritizing action items efficiently.
Smart Meeting Summarization and Action Item Detection is an AI-powered system designed to automate meeting analysis by generating concise summaries and extracting important action items from virtual and hybrid meetings. The system converts meeting recordings into structured textual insights using advanced Artificial Intelligence, Natural Language Processing (NLP), Speech Recognition, and Deep Learning techniques. It integrates OpenAI Whisper for speech transcription, BART and PEGASUS for abstractive summarization, and BERT for entity extraction and action-item detection.
The project aims to reduce manual note-taking efforts, improve productivity, enhance collaboration, and support faster decision-making in professional, educational, and industrial environments.
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
Traditional meeting documentation methods are time-consuming and often fail to capture important discussions, decisions, and tasks accurately. Existing summarization systems struggle with multi-speaker conversations, overlapping speech, noisy audio, and long unstructured discussions. This project addresses these challenges by developing an intelligent automated system capable of generating accurate summaries and extracting actionable tasks with priorities and deadlines.
The architecture integrates Whisper ASR, BART, PEGASUS, and BERT models to generate structured meeting insights.
The project uses the AMI Meeting Corpus dataset containing approximately 140 multi-speaker meetings recorded in English.
The dataset is suitable for meeting summarization, speech recognition, and action-item extraction research.
OpenAI Whisper is used for Automatic Speech Recognition (ASR). It converts meeting audio into text transcripts while handling noisy environments and multi-speaker conversations.
BART is a transformer-based abstractive summarization model that generates concise summaries from long conversational transcripts.
PEGASUS is designed specifically for text summarization tasks and produces highly coherent and context-aware summaries.
BERT with BIO tagging is used for entity extraction and action-item detection. It identifies tasks, deadlines, and assigned persons.
The Smart Meeting Summarization and Action Item Detection system successfully automates the process of meeting transcription, summarization, and task extraction using AI and NLP technologies. By integrating Whisper, BART, PEGASUS, and BERT models, the system generates accurate summaries, extracts actionable insights, and prioritizes tasks effectively. The project improves productivity, reduces manual effort, and provides a scalable solution for modern meeting management.
4. Neural Network Backbone, Layer Specifications & Loss Formulations
Smart Meeting Summarization and Action Item Detection is an AI-based system that converts meeting recordings into concise summaries and extracts tasks, deadlines, and assignees. Using Whisper, BART, PEGASUS, and BERT, it improves productivity by automating meeting analysis and prioritizing action items efficiently.
📝 MeetNote — Video & Meeting Intelligence An autonomous platform that converts meetings, video presentations, and voice recordings into structured executive notes, action deliverables, key decisions, and searchable knowledge. ✨ Capabilities 🎥 Dual Input Sources : Process audio/video directly from YouTube URLs or local media uploads ( .mp3 , .wav , .m4a , .mp4 , .mkv , .webm ). 🎙️ Dual Transcription Engines : OpenAI Whisper & Faster-Whisper (INT8) for ultra-fast local decoding. Instant YouTube Caption Extraction (~2s) for multi-hour recordings. Sarvam AI ( saaras:v2.5 ) with parallel workers for Hindi / Hinglish translation. 📋 Executive Briefing : Structured synthesis powered by Mistral AI ( open-mistral-7b ). 🎯 Action Deliverables & Alignments : Extract tasks, assigned owners, deadlines, and key decisions. 💬 Semantic Q&A : In-memory ephemeral ChromaDB vector search & LangChain LCEL RAG. 📄 Export Reports : Instant export to PDF Reports and Markdown Notes . 🚀 Cloud Deployable : 1-click deploy to Render (FastAPI or Streamlit) with zero storage leaks. 🛠️ Tech Stack Frontend : Streamlit Backend API : FastAPI & Uvicorn LLM Orchestration : LangChain LCEL ( langchain-core , langchain-mistralai ) LLM Provider : Mistral AI ( open-mistral-7b ) Vector Store & Embeddings : ChromaDB (Ephemeral In-Memory) & HuggingFace ( all-MiniLM-L6-v2 ) STT Engines : OpenAI Faster-Whisper, YouTube Transcript API, Sarvam AI Media Engine : yt-dlp , pydub , static-ffmpeg PDF Generation : ReportLab 🚀 Quickstart 1. Clone & Setup git clone https://github.com/Raj-Rathod-Ai/MeetNotes.git cd MeetNotes # Create virtual environment python -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate # Install dependencies pip install -r requirements.txt 2. Configure Environment Variables Create a .env file in the root directory: MISTRAL_API_KEY = your_mistral_api_key SARVAM_API_KEY = your_sarvam_api_key SARVAM_STT_MODEL = saaras:v2.5 MISTRAL_MODEL = open-mistral-7b HF_HUB_DISABLE_SYMLINKS_WARNING = 1 3. Run Web App or API # Run Streamlit UI streamlit run app.py # Or Run FastAPI REST Backend uvicorn api:app --reload --port 8000 Open http://localhost:8501 in your browser. ☁️ Deployment on Render This repository includes pre-configured render.yaml blueprints: Connect your GitHub repository to Render . Set your environment variables ( MISTRAL_API_KEY , SARVAM_API_KEY ). Deploy! Automatic ephemeral audio purging keeps disk usage at 0 MB .
An autonomous platform that converts meetings, video presentations, and voice recordings into structured executive notes, action deliverables, key decisions, and searchable knowledge.
git clone https://github.com/Raj-Rathod-Ai/MeetNotes.git cd MeetNotes # Create virtual environment python -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate # Install dependencies pip install -r requirements.txt
MISTRAL_API_KEY = your_mistral_api_key SARVAM_API_KEY = your_sarvam_api_key SARVAM_STT_MODEL = saaras:v2.5 MISTRAL_MODEL = open-mistral-7b HF_HUB_DISABLE_SYMLINKS_WARNING = 1
# Run Streamlit UI streamlit run app.py # Or Run FastAPI REST Backend uvicorn api:app --reload --port 8000
MeetNote is an AI-powered meeting intelligence system using Deep Learning, NLP, LLMs, and RAG. It combines Whisper/Sarvam AI for speech recognition, Mistral AI for summarization and insight extraction, HuggingFace embeddings, ChromaDB vector search, and RAG-based semantic Q&A to turn meetings into searchable knowledge.
Meetings play a critical infrastructural role in coordinating work. The recent surge of hybrid and remote meetings in computer-mediated spaces has led to new problems (e.g., more time spent in less engaging meetings) and new opportunities (e.g., automated transcription/captioning and recap support). Advances in dialogue summarization offer the potential for improving post-meeting experiences, but fixed-length summaries often fail to meet diverse needs, such as quick overviews or detailed insights. To address these gaps, we use cognitive science and discourse theories to conceptualize two recap designs: important highlights and a structured, hierarchical minutes view, targeting complementary recap needs. We operationalize these representations into high-fidelity prototypes using dialogue summarization. Finally, we evaluate the representations’ effectiveness with seven users in the context of their work meetings at Microsoft. Our results show both recap types are valuable in different contexts, enabling collaboration through discussions and consensus-building. Exploring the meaning of users adding, editing, and deleting from recaps suggests varying alignment for using these actions to improve AI-recap. Our design implications, such as incorporating organizational artifacts (e.g., linking presentations) in recaps and personalizing context, advance the discourse of effective recap designs for organizational work and support past results from cognition studies.
With the rise in geographically dispersed teams, the COVID-19 pandemic, and shifts towards remote work, organizations are increasingly adopting online web conferencing software to hold meetings ( Teevan et al., 2022 ; Yankelovich et al., 2004 ) . The wide adoption of technically mediated meetings allows designers and researchers to apply technical approaches to support these new ways of working.
5. Training Protocol, Convergence Behavior & Experimental Evaluation
In this paper, we consider technological support for meeting recap – systems that aid in the capture of knowledge, decisions, and action items conveyed in a meeting for asynchronous review and engagement. A meeting recap is essential for preserving meeting content for attendees and non-attendees alike ( Whittaker et al., 2008 ) . Traditional practices for capturing meeting content, such as paper and electronic note-taking, are often informal and burdensome on meeting participants ( Kalnikaite et al., 2012 ) . Meeting content browsers provide a mix of audio, video, and ASR-generated transcripts with bookmark-style annotations ( Whittaker et al., 2008 ; Geyer et al., 2005 ) helping users in referencing information from recordings but do not eliminate the manual effort of searching and interpreting meeting content ( Arons, 1992 ; Moran et al., 1997a ) . This also limits asynchronous participation – opportunities for non-attendees to engage with the meeting content and with attendees post-meeting – due to the lack of context ( Kalnikaite et al., 2012 ) .
Recent advances in dialogue summarization in NLP ( Lewis et al., 2020 ) can generate contextual summaries from transcripts that can better support meeting recap goals. However, directly applying dialogue summarization on meeting transcripts to generate a fixed-length summary may not have the right structure and information to support diverse goals such as sharing knowledge, achieving consensus, negotiating deals, and deliberations ( Tuggener et al., 2021 ; Allen et al., 2015 ) . For example, the same summary may be too verbose for participants needing quick highlights and too concise for those requiring in-depth context. Thus, an effective recap requires deciding what and how much should be summarized and what structure the recap should take to cater to these often diverging meeting goals. Unlike summaries of short dialogues, meetings have rich organizational context ( Whittaker et al., 2008 ; Kalnikaitundefined and Whittaker, 2008 ) . Hence, potential recap designs require expensive in-context evaluations of real meetings instead of inexpensive third-party crowdsourced evaluations ( Smith et al., 2020 ; Star and Ruhleder, 1994 ; Schaekermann et al., 2018 ) .
To address this gap in identifying effective recap designs and how they can support organizational work, we answer the following research questions.
RQ1: What key structures and information items should be captured in a meeting recap to serve users’ needs?
RQ2: What benefits and challenges do meeting attendees get for their personal and group work from these key structures and information items in the meeting recap?
RQ3: What do meeting attendees’ post-meeting interactions with the recap mean for aligning AI-generated meeting recap with participants’ needs?
Building on CSCW scholarship of exploring technological support for complex work ( Knoll et al., 2022 ) through in-context evaluations of experimental high-fidelity designs, to answer our research questions, we 1) Use design rationales supported by theories from cognitive science and discourses ( Zacks et al., 2001 ) to conceptualize two recap designs ( Yang et al., 2019a ) that serve diverse recap needs – “highlights”, that represent key moments in meetings for quick takeaways, and “hierarchical”, that represents a meeting as chronological topic-focused discussions for sharing knowledge ( Kalnikaitundefined and Whittaker, 2008 ; Nathan et al., 2012 ) , 2) Leverage the concept of experimental prototype experiences from HCI ( Yang et al., 2019a ) to create high-fidelity operationalization our proposed recap designs using dialogue summarization, and 3) Perform preliminary task-based evaluations with seven information workers at Microsoft to explore the benefits and challenges when attendees use the recaps to support their work and collaborations. To address the scarcity of available meeting datasets for training ( Rennard et al., 2023 ) and the need for personalization, we also seek to understand how user interactions could be used to improve the alignment ( Yang, 2017 ) of AI-generated recap models (see Figure 1 ).
Our in-context evaluations indicate that “highlights” and “hierarchical” representations serve complimentary recap needs. While both recaps provided comprehensible summaries, participants preferred highlights for quick overviews and sharing key decisions, and hierarchical for understanding discussions and sharing knowledge ( Geyer et al., 2005 ) . Participants preferred personalization in selecting relevant important notes for the highlights recap or personalized markers to support navigation in the hierarchical recap. Beyond personal work, participants’ desire to use the recap as a collaborative document for consensus and knowledge work indicates the potential for meeting recap as an organizational artifact supporting articulation work ( Schmidt and Bannon, 1992 ) . Participants indicated consistent reasons to add or edit summaries but many diverging reasons to delete summaries, indicating the nuances in user interactions for improving AI-generated summaries. These implications are grounded in the existing literature of increasing alignment of Human-AI models ( Yang et al., 2019b ) .
We contribute a (theoretically backed ( Zacks et al., 2001 ; Mann and Thompson, 1987 ; Grosz and Sidner, 1986 ) ) foundation for designing meeting recap experiences using automated summarization technologies that serve the practical needs of meeting attendees and non-attendees (examined in the context of work) and its role in supporting organizational work. Our work also discusses specific implications of using UX interactions with recap designs for aligning AI-recap with user’s personal recap needs.
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)