Four conference and two workshop papers were accepted at the International Conference on Learning Representations (ICLR 2024)

ICLR24 Main Conference

Image Inpainting via Tractable Steering of Diffusion Models

Authors: Anji Liu, Mathias Niepert, and Guy Van den Broeck

Summary: Diffusion models are the current state-of-the art for generating photorealistic images. However, controlling the sampling process for constrained image generation tasks such as inpainting remains challenging since exact conditioning on such constraints is intractable. While existing methods use various techniques to approximate the constrained posterior, this paper proposes to exploit the ability of Tractable Probabilistic Models (TPMs) to exactly and efficiently compute the constrained posterior, and to leverage this signal to steer the denoising process of diffusion models. Specifically, this paper adopts a class of expressive TPMs termed Probabilistic Circuits (PCs). Building upon prior advances, we further scale up PCs and make them capable of guiding the image generation process of diffusion models. Empirical results suggest that our approach can consistently improve the overall quality and semantic coherence of in painted images across three natural image datasets (i.e., CelebA-HQ, ImageNet, and LSUN) with only ~10% additional computational overhead brought by the TPM.

A. Liu, M. Niepert, and G. den Broeck. Proceedings of the International Conference on Learning Representations(ICLR 2024), May 7-11, 2024, Austria, ICLR, (May 2024)

Near-optimal solutions of constrained learning problems

Authors: Juan Elenter, Luiz F. O. Chamon, and Alejandro Ribeiro

Summary: As ML becomes the core technology underlying critical applications, it becomes increasingly important to develop ML methods that excel at their main task and adhere to requirements such as fairness and robustness. Since virtually all ML models are trained using empirical risk minimization, a natural way to impose requirements is to add constraints to these optimization problems explicitly. Despite their non-convexity, recent works have shown that dual algorithms can be used to tackle these constrained learning tasks with generalization guarantees. Yet, though these algorithms converge in objective value, they do not necessarily provide feasible solutions. Doing so requires randomizing over all iterates, which is impractical in virtually any modern application. Nevertheless, using the last iterate has been observed to perform well in practice. This work addresses this gap between theory and practice by showing that the last iterates have small constraint violations for typical constrained learning tasks. This completes the characterization of dual learning algorithms, shedding light on prior empirical successes.

J. Elenter, L. Chamon, and A. Ribeiro. Proceedings of the International Conference on Learning Representations(ICLR 2024), May 7-11, 2024, Austria, ICLR, (May 2024)

Probabilistically Rewired Message-Passing Neural Networks

Authors: Chendi Qian, Andrei Manolache, Kareem Ahmed, Zhe Zeng, Guy Van den Broeck, Mathias Niepert, and Christopher Morris.

Summary: In our paper, we present Probabilistically Rewired Message-Passing Neural Networks (PR-MPNNs), a variant of graph neural networks designed for processing graph-structured data, such as molecular structures in chemistry. PR-MPNNs are specifically developed to address issues like information bottlenecks and the inability to pass messages effectively over longer ranges, which are common in traditional MPNNs. By adaptively modifying connections within the graph, PR-MPNNs significantly improve their capacity to handle complex structures. Our results demonstrate that PR-MPNNs effectively overcome these limitations, enhancing performance in several application domains, particularly on molecular data.

C. Qian, A. Manolache, K. Ahmed, Z. Zeng, G. den Broeck, M. Niepert, and C. Morris. Proceedings of the International Conference on Learning Representations(ICLR 2024), May 7-11, 2024, Austria, ICLR, (May 2024)

Tractable Probabilistic Graph Representation Learning with Graph-Induced Sum-Product Networks

Authors: Federico Errica and Mathias Niepert

Summary: Current graph machine learning models struggle with overconfident predictions and missing graph data, and cannot answer “what if” queries about the nodes. To overcome these challenges, we introduce graph-induced sum-product networks (GSPNs). A new probabilistic framework for graph representation learning, GSPNs can tractably answer probabilistic queries defined on the graph. Inspired by computational trees in message-passing neural networks, GSPNs offer the benefits of deep graph networks with the added advantages of a purely probabilistic model. The empirical results show GSPNs' competitiveness in scarce supervision scenarios, handling missing data and graph classification compared to popular neural models. The proposed framework bridges the gap between probabilistic and neural models for graph-structured data, increases the trustworthiness of learning systems, and enables more efficient handling of incomplete information.

F. Errica, and M. Niepert. Proceedings of the International Conference on Learning Representations(ICLR 2024), May 7-11, 2024, Austria, ICLR, (May 2024)

Workshops

Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent PDEs

Authors: Jan Hagnberger, Marimuthu Kalimuthu, Daniel Musekamp, Mathias Niepert

Venue:  Workshop on AI for Differentiable Equations in Science at ICLR

Abstract: Neural Operators are a recent class of data-driven models for learning solutions to Partial Differential Equations (PDEs). Traditionally, these models are trained in an autoregressive fashion using data collected at discrete time points in the evolution of the PDE. This setup gives rise to two problems: (i) poor temporal generalization due to error accumulation and (ii) poor zero-shot super-resolution capabilities. To address these issues, we propose Vectorized Conditional Neural Fields (VCNeF), a general framework that utilizes transformers and implicit neural representations to efficiently solve time-dependent PDEs of varying coefficients. A comprehensive evaluation of VCNeF on the challenging 1D and 2D PDEs from PDEBench demonstrates the superiority of our model over four state-of-the-art baselines. Furthermore, our proposed model achieves faster inference and generalizes better to unseen PDE parameters than the compared models.

J. Hagnberger, M. Kalimuthu, D. Musekamp, and M. Niepert. Proceedings of the AI4DifferentialEquations in Science workshop at ICLR 2024, May 7-11, 2024, Austria, ICLR, (May 2024)

Energy Minimizing-based Token Merging for Accelerating Transformers

Authors: Hoai-Chau Tran, Duy Minh Ho Nguyen, Manh-Duy Nguyen, Ngan Hoang Le, Binh T. Nguyen

Venue: Practical ML for Low Resource Settings Workshop at ICLR

Abstract: Model compression has been an active research field that has been used to reduce the size and complexity of the model. In a recent noteworthy study, ToMe and its variants utilize the Bipartite Soft Matching (BSM) algorithm in which tokens representing patches in an image are split into two sets, and top-k similar tokens from one set are merged. This approach utilizes pre-trained weights, enhances speed, and reduces memory usage. However, these algorithms have some drawbacks. First, the choice of a token-splitting strategy significantly influences algorithm performance since tokens in one set can only perceive tokens in the other set, leading to mis-merging issues. Furthermore, although ToMe is effective in the initial layers, it becomes increasingly problematic in deeper layers as the number of tokens diminishes because of damaged informative tokens. To address these limitations, rather than relying on specific splitting strategies like BSM, we propose a new algorithm called PiToMe. Specifically, we prioritize the protection of informative tokens using an additional factor called energy score. In experiments, PiToMe achieved up to a 50% memory reduction while exhibiting superior off-the-shelf performance on image classification ( keeping a 1.71% average performance drop compared to 2.6% for ToMe) and image-text retrieval (1.35% average performance drop compared to 6.89% for ToMe) compared to previous BSM-based approaches dependent solely on token similarity.

H. Tran, D. Nguyen, M. Nguyen, N. Le, and B. T. Nguyen. Proceedings of Practical ML for Low Resource Settings in Science workshop at ICLR 2024, May 7-11, 2024, Austria, ICLR, (May 2024)