Publications

2024

# Image Generation # Diffusion # Prompt Optimization

TIPO: Text to Image with Text Presampling for Prompt Optimization

Shih-Ying Yeh, Sang-Hyun Park, Giyeong Oh, Min Song, Youngjae Yu

# Embodied AI # Robotics # Navigation

CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction

Suhwan Choi, Yongjun Cho, Minchan Kim, Jaeyoon Jung, Myunchul Joe, Yubeen Park, Minseo Kim, Sungwoong Kim, Sungjae Lee, Hwiseong Park, Jiwan Chung, Youngjae Yu

Neurips2024

# Multimodal # Creative AI

Towards Visual Text Design Transfer Across Languages

Yejin Choi, Jiwan Chung, Sumin Shim, Giyeong Oh, Youngjae Yu

Neurips2024

# NLP # AI Safety # LLM # Jailbreaking # Alignment

WILDTEAMING at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models

Liwei Jiang, Kavel Rao, Seungju Han, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Marteen Sap, Yejin Choi, Nouha Dziri

Neurips2024

# NLP # AI Safety # LLM # Moderation

WILDGUARD: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs

Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, Nouha Dziri

EMNLP2024

# Multimodal # Ambiguity

Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!

Jiwan Chung, Seungwon Lim, Jaehyun Jeon, Seungbeen Lee, Youngjae Yu

# 3D # Speech # Facial expression

DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation

Jisoo Kim, Jungbin Cho, Joonho Park, Soonmin Hwang, Da Eun Kim, Geon Kim, Youngjae Yu

Arxiv

# Image Generation # Diffusion # Personalization

Layout-and-Retouch: A Dual-stage Framework for Improving Diversity in Personalized Image Generation

Kangyeol Kim, Wooseok Seo, Sehyun Nam, Bodam Kim, Suhyeon Jeong, Wonwoo Cho, Jaegul Choo, Youngjae Yu

Arxiv

ECCV2024

# Multimodal # Video LMM # Preference

ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos

Hyolim Kang, Jeongseok Hyun, Joungbin An, Youngjae Yu, Seon Joo Kim

Arxiv

EMNLP2024 (findings)

# NLP # Psychological Counseling # Dialogue

CACTUS: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory

Suyeon Lee, Sunghwan Kim, Minju Kim, Dongjin Kang, Dongil Yang, Harim Kim, Minseok Kang, Dayi Jung, Min Hee Kim, Seungbeen Lee, Kyoung-Mee Chung, Youngjae Yu, Dongha Lee, Jinyoung Yeo

Arxiv

EMNLP2024 (findings)

# Multimodal # Fact checking # Misinformation

How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models

Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu, Yonatan Bisk, Yejin Choi, Saadia Gabriel

Arxiv

# Multimodal # Video LMM # Preference

i-SRT: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective Judgment

Daechul Ahn, Yura Choi, San Kim, Youngjae Yu, Dongyeop Kang, Jonghyun Choi

Arxiv

EMNLP2024 (Oral)

# Multimodal Understanding # Visual Reasoning

Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding

Jiwan Chung, Sungjae Lee, Minseo Kim, Seungju Han, Ashkan Yousefpour, Jack Hessel, Youngjae Yu

Arxiv

# Computer Vision # Scalp Diagnosis # Image Translation

Scalp Diagnostic System With Label-Free Segmentation and Training-Free Image Translation

Youngmin Kim, Saejin Kim, Hoyeon Moon, Youngjae Yu, Junhyug Noh

Arxiv

# NLP # Personality # Psychometrics

Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics

Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Jiwan Chung, Minju Kim, Yeonsoo Lee, Dongha Lee, Jinyoung Yeo, Youngjae Yu

Arxiv

ACL2024 (Oral)

# Multimodal # RLAIF

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, Jonghyun Choi

Arxiv

ACL2024

# NLP # Reward Modeling

Aligning Large Language Models by On-Policy Self-Judgment

Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, Youngjae Yu

Arxiv

ACL2024

# NLP # Conversation # Recommendation

Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset

Minjin Kim, Minju Kim, Hana Kim, Beong-woo Kwak, Soyeon Chun, Hyunseo Kim, SeongKu Kang, Youngjae Yu, Jinyoung Yeo, Dongha Lee

Arxiv

ACL2024 (Outstanding)

# NLP # Conversation

Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation

Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, Jinyoung Yeo

Arxiv

# korean-LLM # Naver

HyperCLOVA X Technical Report

Jiwan Chung, Sangkyu Lee, Youngjae Yu contributed.

Arxiv

EMNLP2024

# NLP # Reasoning # Code Generation

Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models

Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu-iunn Ong, Beong-woo Kwak, Seonghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, Jinyoung Yeo

Arxiv

NAACL2024

# multimodal # Commonsense # Video Understaning

SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models

Hyun Lee, Kim Sung-Bin, Seungju Han, Youngjae Yu, Tae-Hyun Oh

Arxiv

ICLR2024

# Text-to-Image # PEFT

Navigating Text-To-Image Customization:From LyCORIS Fine-Tuning to Model Evaluation

Shin-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, Yanmin Gong

Arxiv

2023

Neurips2023

# multimodal # Commonsense

Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Jae Sung Park, Jack Hessel, Khyathi Chandu, Paul Pu Liang, Ximing Lu, Youngjae Yu, Qiuyuan Huang, Peter West, Jianfeng Gao, Ali Farhadi, Yejin Choi, Qiuyuan Huang

Arxiv

EMNLP2023 (Oral)

# multimodal # Commonsense # Social Norm

Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms

Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Jiwan Chung, Yejin Son, Yejin Choi, Youngjae Yu

Arxiv

EMNLP2023

# multimodal

VLIS: Unimodal Language Models Guide Multimodal Language Generation

Jiwan Chung, Youngjae Yu

Arxiv

EMNLP2023 (Outstanding)

# NLP # Commonsense Knowledge # Distillation # Dialog

SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization

Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, Yejin Choi

Arxiv

EMNLP2023

# NLP # Commonsense Knowledge # Dialog

Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents

Hyungjoo Chae, Yongho Song, Kai Tzu-iunn Ong, Taeyoon Kwon, Minjin Kim, Youngjae Yu, Dongha Lee, Dongyeop Kang, Jinyoung Yeo

Arxiv

BMVC2023

# VideoQA # multimodal

Long Story Short: a Summarize-then-Search Method for Long Video Question Answering

Jiwan Chung, Youngjae Yu

Arxiv

ICRA2024

# Robotics # NLP # uncertainty estimation

CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents

Jeongeun Park, Seungwon Lim, Joonhyung Lee, Sangbeom Park, Minsuk Chang, Youngjae Yu, Sungjoon Choi

Arxiv

ACL2023

# NLP # Chain-of-thought

Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step

Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, Yejin Choi

Arxiv

NeurIPS2023

# Computer Vision # NLP # augmentation # interleaved image+text

Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text

Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi

Arxiv

ICCV2023

# Computer Vision # NLP # Conversation # Visual Context

CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos

Seungju Han, Jack Hessel, Nouha Dziri, Yejin Choi, Youngjae Yu

Arxiv

CVPR2023

# NLP # Computer Vision # Reinforcement Learning # Zero-shot

Fusing Pre-trained Language Models with Multimodal Prompts through Reinforcement Learning

Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, Jae Sung Park, Ximing Lu, Prithviraj Ammanabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, Yejin Choi

Arxiv

ICRA2023

# Computer Vision # Robotics # Object Detection

Zero-shot Active Visual Search (ZAVIS): Intelligent Object Search for Robotic Assistants

Jeongeun Park, Taerim Yoon, Jejoon Hong, Youngjae Yu, Matthew Pan, Sungjoon Choi

Arxiv

Publications

2024

TIPO: Text to Image with Text Presampling for Prompt Optimization

CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction

Towards Visual Text Design Transfer Across Languages

WILDTEAMING at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models

WILDGUARD: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs

Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!

DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation

Layout-and-Retouch: A Dual-stage Framework for Improving Diversity in Personalized Image Generation

ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos

CACTUS: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory

How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models

i-SRT: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective Judgment

Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding

Scalp Diagnostic System With Label-Free Segmentation and Training-Free Image Translation

Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

Aligning Large Language Models by On-Policy Self-Judgment

Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset

Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation

HyperCLOVA X Technical Report

Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models

SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models

Navigating Text-To-Image Customization:From LyCORIS Fine-Tuning to Model Evaluation

2023

Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms

VLIS: Unimodal Language Models Guide Multimodal Language Generation

SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization

Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents

Long Story Short: a Summarize-then-Search Method for Long Video Question Answering

CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents

Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step

Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text

CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos

Fusing Pre-trained Language Models with Multimodal Prompts through Reinforcement Learning

Zero-shot Active Visual Search (ZAVIS): Intelligent Object Search for Robotic Assistants

2022

NeuroLogic A* esque Decoding: Constrained Text Generation with Lookahead Heuristics

Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer

MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

PROSOCIALDIALOG:A Prosocial Backbone for Conversational Agents

2021

Dual Compositional Learning in Interactive Image Retrieval

Parameter Efficient Multimodal Transformers for Video Representation Learning

Self-Supervised Learning of Compressed Video Representations

Transitional Adaptation of Pretrained Models for Visual Storytelling

Pano-AVQA: Grounded Audio-Visual Question Answering on 360° Videos

Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning

MERLOT: Multimodal Neural Script Knowledge Models

2020

Character Grounding and Re-Identification in Story of Videos and Text Descriptions

2018

A Deep Ranking Model for Spatio-Temporal Highlight Detection from a 360° Video

A Memory Network Approach for Story-based Temporal Summarization of 360° Videos

A Joint Sequence Fusion Model for Video Question Answering and Retrieval

2017

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering

Supervising Neural Attention Models for Video Captioning by Human Gaze Data

2016

TimesVector_A Vectorized Clustering Approach to the Analysis of Time Series Transcriptome Data from Multiple Phenotypes