In a handful of cases, where the lack of imperative, stack-based, mutable variables in Clojure may make the code not perform as well as Java, we may need to evaluate alternatives to make it faster. I would advise you to consider writing such code directly in Java for better performance.
Another consideration is to use native OS capabilities, such as memory-mapped buffers (http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html) or files and unsafe operations (http://highlyscalable.wordpress.com/2012/02/02/direct-memory-access-in-java/). Note that unsafe operations are potentially hazardous and not recommended in general. Such times are also an opportunity to consider writing performance-critical pieces of code in C or C++ and then access them via the Java Native Interface (JNI).