The following are examples of our current research projects.
Estimation of Information-Theoretic Quantities using Generative Models and Variational Bounds
We develop tools to numerically estimate the fundamental limits, such as rate-distortion trade-offs and capacity expressions, for data compression and communication. This is crucial for identifying the gap between current algorithms and theoretical limits. The core challenge lies in jointly estimating information-theoretic quantities and solving optimization problems for high-dimensional data sources. Our approach leverages generative models and variational bounds, enabling evaluation across a wide range of information quantities beyond traditional rate-distortion and capacity frameworks.
Related Work
Coding for Task-Aware Distributed Compression
We develop a framework for constructing source codes tailored to distributed compression and diverse task/performance objectives, with applications in computer vision, robotics, and communication channels. Our approach leverage the use of generative models which are based on the idea that high-dimensional data can be represented in a low-dimensional representation, making them naturally suited for data compression. Our problem is more complex and interesting than conventional generative model-driven compression for two main reasons: we need to (a) learn complex correlation structures across multiple data sources (e.g., two camera views) and (b) construct algorithms that can use these low-dimensional representations to make distributed compression effective for specific tasks and/or function computation. The novelty of our approach lies in extracting low-dimensional representations and developing associated generative models combined with traditional analytical algorithms. Our recent work supports this hybrid method, where we combine the distributed principal component analysis together with neural feature learning.
Related Work
-
- Task-aware Distributed Source Coding under Dynamic Bandwidth
Authors: Po-Han Li*, Sravan Ankireddy*, Ruihan Zhao, Hossein Mahjoub, Ehsan Pari, Ufuk Topcu, Sandeep Chinchali, Hyeji Kim [NeurIPS 2023] - Neural Distributed Source Coding
Authors: Jay Whang*, Alliot Nagle*, Anish Acharya, Hyeji Kim, Alex Dimakis [IEEE JSAIT 2024]
- Task-aware Distributed Source Coding under Dynamic Bandwidth
Coding for Complex Communication Systems
For complex communication systems, such as those involving high-dimensional channels, we employ deep learning to optimize channel codes. Our key novelty lies in limiting and guiding the search using information-theoretic insights and developing new insights based on learning-based results.
Related Work
-
- Channel Coding via Machine Learning (Book chapter)
Author: Hyeji Kim [Machine Learning and Wireless Communications by Y Eldar, A Goldsmith, D Gunduz, V Poor] - DeepPolar: Inventing Nonlinear Large-Kernel Polar Codes via Deep Learning
Authors: Ashwin Hebbar*, Sravan Ankireddy*, Hyeji Kim, Sewoong Oh, Pramod Viswanath [ICML 2024] - Neural Cover Selection for Image Steganography
Authors: Karl Chahine, Hyeji Kim [NeurIPS 2024]
- Channel Coding via Machine Learning (Book chapter)
Machine Learning for Wireless Communications
We employ machine learning techniques to optimize wireless systems, including CSI compression, feedback mechanisms, and robust channel coding and modulation. Generative models are used to simulate high-dimensional data for training downstream task models, enabling advancements such as intent- and environment-specific communication algorithms tailored to real-world scenarios.
As a concrete example, we address the problem of modeling high-dimensional channels using generative models. Beyond being of fundamental academic interest, this problem is crucial for learning communication algorithms, as training these algorithms requires a large volume of channel measurements, which are expensive to collect. We model location-specific channel distributions using conditional diffusion models to capture the strong location dependency of wireless channels. We demonstrate that these models generate realistic (location, channel) data pairs for unseen locations, enhancing the training of communication algorithms tailored to specific environments.
Related Work
Establishing Theoretical Framework for Deep Learning
We apply principles of information theory to machine learning by constructing theoretical frameworks that uncover fundamental limits and provide deep insights into learning approaches. For example, we formalize the problem of prompt compression for large language models (LLMs) and propose a unified framework for token-level compression methods that create prompts for black-box models. By deriving the distortion-rate function as a linear program and developing an efficient algorithm to compute it, we establish a baseline for evaluating prompt compression methods. Notably, we identify a substantial gap between existing methods and the theoretical limit and introduce a variable-rate prompt compression method to bridge this gap.
Related Work
-
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
Authors: Alliot Nagle*, Adway Girish*, Marco Bondaschi, Michael Gastpar, Ashok V. Makkuva, Hyeji Kim [NeurIPS 2024]
- Local to Global: Learning Dynamics and Effect of Initialization for Transformers
Authors: Ashok V. Makkuva, Marco Bondaschi, Chanakya Ekbote, Adway Girish, Alliot Nagle, Hyeji Kim, Michael Gastpar [NeurIPS 2024]
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models