I am currently a Computer Science PhD candidate, working under the guidance of Prof. Peter Richtárik. Prior to this, I completed my Master’s degree under the supervision of Prof. Mohamed Elhoseiny at KAUST in December 2021. I earned my Bachelor of Engineering with honors from Xi’an Jiaotong University (XJTU) in June 2019. I have had the opportunity to intern at several leading research institutions, including SonyAI, Vector Institute, Tencent AI Lab, CMU Xulab, NUS CVML Group, and SenseTime Research. My primary research interest lies in centralized and federated LLM compression. My work is highly interconnected, featuring significant projects such as the LLM post-training compression algorithm PV-Tuning (NeurIPS Oral), with more on the way; communication-efficient federated learning methods CohortSqueeze (NeurIPS-W Oral), FedP3 (ICLR), and EF-BV (NeurIPS); and multimodal language model projects DACZSL (ICCVW), HGR-Net (ECCV), and VisualGPT (CVPR).

I am actively seeking research internship and full-time opportunities. If you are interested in my profile, please feel free to contact me.

Research Interests

LLM Compression: Post-training Pruning/Quantization, Extreme Quantization, Efficient Pretraining, and Fine-Tuning
Federated Learning: Personalization, Sparsity, Data/Model Heterogeneity, Federated LLMs
Distributed Optimization: Distributed SGD, Variance Reduction, Compression

What’s New

New!! 2024.10. Our paper Cohort Squeeze has been accepted by NeurIPS FL@FM Workshop as an Oral!
New!! 2024.09. Our paper PV-Tuning has been accepted by NeurIPS 2024 as an Oral (acceptance rate 0.4%, 61/15671)!
2024.08. Invited as a reviewer for ICLR 2025.
2024.07. Invited as a reviewer for IEEE Transactions on Signal Processing.
New!! 2024.06. Our paper Cohort Squeeze is available on arXiv!
New!! 2024.06. Our paper Sparse-ProxSkip is available on arXiv!
New!! 2024.05. Our paper PV-Tuning is available on arXiv! Code has been released.
2024.05. Reviewer for NeurIPS 2024.
2024.05. Attending ICLR 2024 in Vienna, Austria.
New!! 2024.04. Passed the PhD proposal defense with the title “Exploring Real-World Challenges in Federated Learning: Personalization, Sparsity, and Scalability”!
2024.04. Our paper FedP3 is available on arXiv. Code has been released.
2024.03. New paper “FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models” is available on arXiv.
2024.03. An improved version of DACZSL is on arXiv.
2024.01. Invited as a reviewer for ECCV 2024.
2024.01. Our paper FedP3 has been accepted by ICLR24!
2023.11. Reviewer for CVPR 2024.
2023.09. Reviewer for AISTATS 2024.
2023.09. Our paper DACZSL has been accepted by the ICCV23 OOD-CV workshop.
2023.08. Reviewer for Computer Vision and Image Understanding (CVIU).
2023.07. Our paper IGCZSL has been accepted by ICCV 2023!
2023.07. Program Committee member for AAAI 2024.
2023.06. Research Internship at SonyAI.
2023.06. Reviewer for IJCV.
2023.05. Reviewer for NeurIPS 2023, WACV 2024, BMVC 2023.
2023.05. Our paper “Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learning” is available on arXiv. Code is available here.
2023.03. Reviewer for ICCV 2023, IJCV, T-PAMI.
2022.12. Attending NeurIPS 2022 in New Orleans, LA.
2022.11. Reviewer for ICLR 2023, CVPR 2023.
2022.10. Reviewer for AISTATS 2023.
2022.09. Two papers (EF-BV and VR-ProxSkip) accepted by NeurIPS 2022!
2022.09. Program Committee member (PC) for AAAI 2023.
2022.08. Serve as an Orientation Leader at KAUST 2022 Fall.
2022.07. Our paper “Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning” is available on arXiv.
2022.07. HGR-Net for large-scale zero-shot learning has been accepted by ECCV 2022! Code is available here.
2022.06. Reviewer for WACV 2023, BMVC 2022.
2022.06. I moved from Building-1 L4 to Building-1 L2.
2022.06. Reviewer for NeurIPS 2022.
2022.06. Continue serving as a Student Ambassador 2022-2023 at KAUST CEMSE!
2022.05. Our paper “EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimization” is available on arXiv！
2022.04. Our paper “Creative Walk Adversarial Networks: Novel Art Generation with Probabilistic Random Walk Deviation from Style Norms” has been accepted by ICCC 2022!
2022.04. Our short paper “Language-Guided Imaginative Walks: Generative Random Walk Deviation Loss for Unseen Class Recognition using Text” has been accepted by CVPR22 L3D-IVU Workshop!
2022.03. HGR-Net for large-scale zero-shot learning is available on arXiv.
2022.03. Our paper VisualGPT is accepted by CVPR22! code has been released.
2022.01-05. Reviewer for ECCV22, CVPR22, ICML22, IJCV, TIP.
2022.02. Teaching Assistant for CS283: Deep Generative Modeling.
2021.12. DACZSL is available on arXiv.
2021.12. Glad to be a PhD student at Optimization and Machine Learning Lab led by Prof. Peter Richtárik!
2021.12. Graduated from KAUST with a Master degree. Thanks a lot to my MS supervisor Prof. Mohamed Elhoseiny and colleagues. Also congratulations to myself!
2021.11. My Master’s thesis is available, homepage.
2021.11. Successfully defended my Master’s thesis!
2021.05. Our unsupervised open-set recognition work has been accepted to ICIP 2021!
2021.04.28. Spotlight talk of CIZSL++ at KAUST AI Initiative. link.
2021.04. Our imaginative walk paper is available on arXiv! –> homepage
2020.12. Our paper VisualGPT is on arXiv. code.
2020.12. Our paper CIZSL++ has been submitted to PAMI. arXiv paper and code are available.
2020.12. Start research internship at Tencent AI Lab.
2020.08. Technical report of Legendre decomposition in machine learning is available on arXiv.
2020.05. Start research at KAUST Vision-CAIR group.
2020.04. Glad to be a MS/PhD student at KAUST.
2020.02. Join CMU Xulab as a remote research intern.

Kai Yi

Research Interests

What’s New