Using spring + jasypt Dynamic Configuration Reading Causing OOM

Recently, we encountered a production service OOM incident. Because the root cause was quite hidden, locating and fixing the issue took nearly a month.

Origin

A month ago, one pod of our production service restarted. After receiving an alert, we checked the pod events and found that the health check failed, so kubelet automatically restarted the pod. Checking the pod logs, no errors were found; checking the request traffic, no spike was detected; checking CPU and memory usage, limits were not exceeded. At that time, we thought it might be an occasional network fluctuation and decided to observe whether it would recur.

Initial Investigation

A few days later, two more pods of the same service also restarted, with exactly the same symptoms. We realized that there was likely a problem, so we carefully examined all monitoring metrics and found that during pod restarts, the heap memory was almost exhausted, the service’s full GC count spiked, GC took several seconds, and old generation objects did not decrease after full GC. This confirmed a memory leak, and the Stop-The-World during full GC caused the service to pause, failing to respond to kubelet health checks, which triggered the restart.

After discovering the memory leak, a heap dump was needed for analysis. However, since the service pods had already restarted, the heap memory usage had returned to normal, making it difficult to locate the issue. We had to let the service run for a while until heap memory usage increased again for further investigation.

Heap Dump Analysis

After another ten days, we checked the service heap memory and found the usage was nearly 80%, with the old generation proportion visibly increasing daily. We performed a heap dump on the pod:

jmap -dump:live,format=b,file=/tmp/heap.hprof 1

Importing the hprof file into IDEA for analysis, we found 28 million ConcurrentHashMap Node objects, occupying 1.48 GB of heap memory: Examining the large objects, we found most were jasypt’s EncryptableMapPropertySourceWrapper: Inspecting the ConcurrentHashMap details, many keys shared the same regionMapping prefix: Searching the project globally, we found the corresponding code:

ApplicationContextUtils.getContext().getEnvironment().getProperty("regionMapping." + val);

This code resides in an SQL interceptor, where val is a field value from SQL. That is, for every SQL field value, this line queries the configuration file for the corresponding config. The problem is that this is only a read operation, so how could it cause data to be written?

We looked at the source code of jasypt’s EncryptableMapPropertySourceWrapper:

package com.ulisesbocchio.jasyptspringboot.wrapper;

import com.ulisesbocchio.jasyptspringboot.caching.CachingDelegateEncryptablePropertySource;
import com.ulisesbocchio.jasyptspringboot.EncryptablePropertyFilter;
import com.ulisesbocchio.jasyptspringboot.EncryptablePropertyResolver;
import com.ulisesbocchio.jasyptspringboot.EncryptablePropertySource;
import org.springframework.core.env.MapPropertySource;
import org.springframework.core.env.PropertySource;

import java.util.Map;

/**
 * @author Ulises Bocchio
 */
public class EncryptableMapPropertySourceWrapper extends MapPropertySource implements EncryptablePropertySource<Map<String, Object>> {

    private final EncryptablePropertySource<Map<String, Object>> encryptableDelegate;

    public EncryptableMapPropertySourceWrapper(MapPropertySource delegate, EncryptablePropertyResolver resolver, EncryptablePropertyFilter filter) {
        super(delegate.getName(), delegate.getSource());
        encryptableDelegate = new CachingDelegateEncryptablePropertySource<>(delegate, resolver, filter);
    }

    @Override
    public void refresh() {
        encryptableDelegate.refresh();
    }

    @Override
    public Object getProperty(String name) {
        return encryptableDelegate.getProperty(name);
    }

    @Override
    public PropertySource<Map<String, Object>> getDelegate() {
        return encryptableDelegate.getDelegate();
    }
}

The getProperty method uses encryptableDelegate of type CachingDelegateEncryptablePropertySource. The “Caching” in the name hints at its behavior. Looking at CachingDelegateEncryptablePropertySource source:

package com.ulisesbocchio.jasyptspringboot.caching;

import com.ulisesbocchio.jasyptspringboot.EncryptablePropertyFilter;
import com.ulisesbocchio.jasyptspringboot.EncryptablePropertyResolver;
import com.ulisesbocchio.jasyptspringboot.EncryptablePropertySource;
import org.springframework.cache.concurrent.ConcurrentMapCache;
import org.springframework.core.env.PropertySource;
import org.springframework.util.Assert;

public class CachingDelegateEncryptablePropertySource<T> extends PropertySource<T> implements EncryptablePropertySource<T> {
    private final PropertySource<T> delegate;
    private final EncryptablePropertyResolver resolver;
    private final EncryptablePropertyFilter filter;
    private final ConcurrentMapCache cache;

    public CachingDelegateEncryptablePropertySource(PropertySource<T> delegate, EncryptablePropertyResolver resolver, EncryptablePropertyFilter filter) {
        super(delegate.getName(), delegate.getSource());
        Assert.notNull(delegate, "PropertySource delegate cannot be null");
        Assert.notNull(resolver, "EncryptablePropertyResolver cannot be null");
        Assert.notNull(filter, "EncryptablePropertyFilter cannot be null");
        this.delegate = delegate;
        this.resolver = resolver;
        this.filter = filter;
        this.cache = new ConcurrentMapCache("encryptablePropertiesCache");
    }

    @Override
    public PropertySource<T> getDelegate() {
        return delegate;
    }

    @Override
    public Object getProperty(String name) {
        return cache.get(name, () -> getProperty(resolver, filter, delegate, name));
    }

    @Override
    public void refresh() {
        cache.clear();
    }
}

As expected, ConcurrentMapCache is used to cache the results of each getProperty call.

jasypt is a common Spring library for encrypting/decrypting configuration, supporting ENC()-formatted encrypted properties to prevent sensitive data exposure. Each read requires decryption, and jasypt caches the decrypted result to avoid unnecessary CPU consumption. Unexpectedly, due to our dynamic reads, this caching mechanism caused a memory leak.

Conclusion

jasypt caches Spring configuration reads. Dynamic reading should be avoided to prevent unbounded cache growth leading to memory leaks. Configuration can be fully loaded into memory first, then retrieved from memory, which prevents similar issues.