Logs show error "java.lang.OutOfMemoryError:unable to create new native thread"

SYMPTOM

When running many processes, each with many threads, as the AtScale or runtime user, the error "java.lang.OutOfMemoryError:unable to create new native thread" may be seen under heavy load.

This error indicates the user has hit the thread limit set at the system level.

RESOLUTION

It is recommended to check the thread limit as the AtScale or runtime user using: # ulimit -u

This issue can be resolved by creating a new file for the runtime user (in the below example, atscale) to increase the thread limit. Using the root user, do as follows:

# vi /etc/security/limits.d/atscale.conf 
atscale   - nofile 65536
atscale   - nproc  65536

This requires a restart of all AtScale services, including the supervisor process.

-- To stop the services:
# /opt/atscale/current/bin/atscale_stop

-- To start the services:
# /opt/atscale/current/bin/atscale_start