Comments on: Turbocharge your string keyed hashmaps http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/?utm_source=rss&utm_medium=rss&utm_campaign=turbocharge-your-string-keyed-hashmaps Dhananjay Nene's opinions on software programming, design, architecture and the internet Sun, 22 Jan 2012 06:53:20 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: David Shay http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-93 David Shay Mon, 28 Apr 2008 11:26:17 +0000 http://blog.dhananjaynene.com/archives/25#comment-93 I have many doubts about your optimization: 1. You should NEVER use new String(String). It forces the JVM to create a new String object. 2. String literals, such as "mykey", are already automatically interned by Java. 3. The call to intern is slow, since it performs a lookup in a hashtable containing ALL String literals. 4. Hashcode is already stored in String, and calculated the first time it is needed (see source code of String.java) I have many doubts about your optimization:
1. You should NEVER use new String(String). It forces the JVM to create a new String object.
2. String literals, such as “mykey”, are already automatically interned by Java.
3. The call to intern is slow, since it performs a lookup in a hashtable containing ALL String literals.
4. Hashcode is already stored in String, and calculated the first time it is needed (see source code of String.java)

]]> By: David Shay http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-403 David Shay Mon, 28 Apr 2008 10:26:17 +0000 http://blog.dhananjaynene.com/archives/25#comment-403 I have many doubts about your optimization:<br>1. You should NEVER use new String(String). It forces the JVM to create a new String object.<br>2. String literals, such as "mykey", are already automatically interned by Java.<br>3. The call to intern is slow, since it performs a lookup in a hashtable containing ALL String literals.<br>4. Hashcode is already stored in String, and calculated the first time it is needed (see source code of String.java) I have many doubts about your optimization:
1. You should NEVER use new String(String). It forces the JVM to create a new String object.
2. String literals, such as “mykey”, are already automatically interned by Java.
3. The call to intern is slow, since it performs a lookup in a hashtable containing ALL String literals.
4. Hashcode is already stored in String, and calculated the first time it is needed (see source code of String.java)

]]> By: Daily del.icio.us for April 20th through April 22nd — Vinny Carpenter’s blog http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-85 Daily del.icio.us for April 20th through April 22nd — Vinny Carpenter’s blog Tue, 22 Apr 2008 14:01:24 +0000 http://blog.dhananjaynene.com/archives/25#comment-85 [...] /var/log/mind » Blog Archive » Turbocharge your string keyed hashmaps - In most most situations the possible universe of keys in the hashmap are known upfront either when writing the code or when starting up the application. If instead of creating hard coded strings or by using various string key parameters from say an XML fi Tags: actionscript, adobe, air, amazon, apple, aws, blog, business, cloudcomputing, Cluster, datacenter, development, ec2, ecma, flash, flex, foss, hashmap, ide, idea, intellij, internet, iptv, iso, java, jetbrains, Linux, mac, microsoft, mxml, networking, norway, ooxml, performance, politics, programming, ria, s3, security, standards, telecom, tips, tools, UVerse, virtualization, virtualpc, vmware, Web2.0, windows, xensource [...] [...] /var/log/mind » Blog Archive » Turbocharge your string keyed hashmaps – In most most situations the possible universe of keys in the hashmap are known upfront either when writing the code or when starting up the application. If instead of creating hard coded strings or by using various string key parameters from say an XML fi Tags: actionscript, adobe, air, amazon, apple, aws, blog, business, cloudcomputing, Cluster, datacenter, development, ec2, ecma, flash, flex, foss, hashmap, ide, idea, intellij, internet, iptv, iso, java, jetbrains, Linux, mac, microsoft, mxml, networking, norway, ooxml, performance, politics, programming, ria, s3, security, standards, telecom, tips, tools, UVerse, virtualization, virtualpc, vmware, Web2.0, windows, xensource [...]

]]> By: Dimitris Andreou http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-84 Dimitris Andreou Mon, 21 Apr 2008 08:49:50 +0000 http://blog.dhananjaynene.com/archives/25#comment-84 If performance is the main issue and you are creating a Symbol class anyway, consider putting the associated values directly in Symbol's fields. That will be as fast as it gets. If performance is the main issue and you are creating a Symbol class anyway, consider putting the associated values directly in Symbol’s fields. That will be as fast as it gets.

]]> By: Khalil http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-81 Khalil Thu, 17 Apr 2008 21:15:00 +0000 http://blog.dhananjaynene.com/archives/25#comment-81 Calling intern can lead to permgen memory exhaustion see this bug issue on Xstream http://jira.codehaus.org/browse/XSTR-395 the fix involves using weak references to get around the cache size issue Calling intern can lead to permgen memory exhaustion see this bug issue on Xstream http://jira.codehaus.org/browse/XSTR-395 the fix involves using weak references to get around the cache size issue

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-80 Dhananjay Nene Thu, 17 Apr 2008 20:34:32 +0000 http://blog.dhananjaynene.com/archives/25#comment-80 @khalil Addressed your concerns in an updated version. @khalil Addressed your concerns in an updated version.

]]> By: Khalil http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-402 Khalil Thu, 17 Apr 2008 20:15:00 +0000 http://blog.dhananjaynene.com/archives/25#comment-402 Calling intern can lead to permgen memory exhaustion see this bug issue on Xstream <a href="http://jira.codehaus.org/browse/XSTR-395">http://jira.codehaus.org/browse/XSTR-395</a> the fix involves using weak references to get around the cache size issue Calling intern can lead to permgen memory exhaustion see this bug issue on Xstream http://jira.codehaus.org/browse/XSTR-395 the fix involves using weak references to get around the cache size issue

]]> By: Khalil http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-79 Khalil Thu, 17 Apr 2008 19:54:26 +0000 http://blog.dhananjaynene.com/archives/25#comment-79 Well to begin with String being mostly an immutable object computing the hashcode once seems a natural and hashCode does cache the calculated value. The reason why fast code is indeed faster is because the compiler interns String literals http://java.sun.com/j2se/1.4.2/docs/api/java/lang/String.html#intern() As to the static HashMap in the symbol class it is I am afraid a memory leak being a cache with no eviction strategy and one that can not be flushed, as a bonus getSymbol is not Thread safe! Static is indeed evil http://gbracha.blogspot.com/2008/02/cutting-out-static.html Well to begin with String being mostly an immutable object computing the hashcode once seems a natural and hashCode does cache the calculated value. The reason why fast code is indeed faster is because the compiler interns String literals http://java.sun.com/j2se/1.4.2/docs/api/java/lang/String.html#intern()
As to the static HashMap in the symbol class it is I am afraid a memory leak being a cache with no eviction strategy and one that can not be flushed, as a bonus getSymbol is not Thread safe! Static is indeed evil http://gbracha.blogspot.com/2008/02/cutting-out-static.html

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-401 Dhananjay Nene Thu, 17 Apr 2008 19:34:32 +0000 http://blog.dhananjaynene.com/archives/25#comment-401 @khalil Addressed your concerns in an updated version. @khalil Addressed your concerns in an updated version.

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-78 Dhananjay Nene Thu, 17 Apr 2008 19:26:00 +0000 http://blog.dhananjaynene.com/archives/25#comment-78 @Dave. To correct myself, I think if String.intern() is called each time a string key is used, one should get similar results without needing a Symbol class. @Dave. To correct myself, I think if String.intern() is called each time a string key is used, one should get similar results without needing a Symbol class.

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-77 Dhananjay Nene Thu, 17 Apr 2008 19:22:07 +0000 http://blog.dhananjaynene.com/archives/25#comment-77 @Jacob. I agree with the non-warming up and the non accounting of symbol map lookup during Symbol construction time. Here's the rationale. Actually I am not so concerned with the timings for the usual (actually I should've called it identical). If a programmer can keep track of ensuring that the strings are identical - either explicitly or by always keeping a cached version of String.intern() result as suggested by Dave then Symbol class isn't really required. The 1% keeps on varying between slightly negative and slightly positive across different runs (basically the additional cost of lookup and the cached value of hashcode are being traded off with each other). Yes the test does not reflect the cost of creation of a symbol. But in most programs, the number of times symbol is created can be made to be much much lower than the number of times it is used for lookups. What you suggest of as a real world example is actually not necessarily a good real world example - since it is quite feasible to ensure that the symbols get created far less often than the getter is called. @Jacob. I agree with the non-warming up and the non accounting of symbol map lookup during Symbol construction time. Here’s the rationale. Actually I am not so concerned with the timings for the usual (actually I should’ve called it identical). If a programmer can keep track of ensuring that the strings are identical – either explicitly or by always keeping a cached version of String.intern() result as suggested by Dave then Symbol class isn’t really required. The 1% keeps on varying between slightly negative and slightly positive across different runs (basically the additional cost of lookup and the cached value of hashcode are being traded off with each other). Yes the test does not reflect the cost of creation of a symbol. But in most programs, the number of times symbol is created can be made to be much much lower than the number of times it is used for lookups. What you suggest of as a real world example is actually not necessarily a good real world example – since it is quite feasible to ensure that the symbols get created far less often than the getter is called.

]]> By: Jacob Hookom http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-76 Jacob Hookom Thu, 17 Apr 2008 19:11:11 +0000 http://blog.dhananjaynene.com/archives/25#comment-76 Looking at your tests, theres a couple issues: 1) you run the 'usual' test first without a JVM warmup-- so that 1% probably shouldn't even be there, I'd even say you'd hit a negative ratio there. 2) You aren't accounting for the Symbol map lookup in your tests and are using a hard reference to the symbols. A real world example of the testSymbol would be to walk through each String key, do a Symbol.get(...) and then do the hashmap lookup. Looking at your tests, theres a couple issues: 1) you run the ‘usual’ test first without a JVM warmup– so that 1% probably shouldn’t even be there, I’d even say you’d hit a negative ratio there. 2) You aren’t accounting for the Symbol map lookup in your tests and are using a hard reference to the symbols. A real world example of the testSymbol would be to walk through each String key, do a Symbol.get(…) and then do the hashmap lookup.

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-75 Dhananjay Nene Thu, 17 Apr 2008 19:04:01 +0000 http://blog.dhananjaynene.com/archives/25#comment-75 @Dave. Have updated the post with the sources to be able to verify my findings. I missed the Strings.intern() part. I think it will certainly make for a simpler Symbol implementation. However the Symbol implementation will still be required. @Dave. Have updated the post with the sources to be able to verify my findings. I missed the Strings.intern() part. I think it will certainly make for a simpler Symbol implementation. However the Symbol implementation will still be required.

]]> By: Khalil http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-400 Khalil Thu, 17 Apr 2008 18:54:26 +0000 http://blog.dhananjaynene.com/archives/25#comment-400 Well to begin with String being mostly an immutable object computing the hashcode once seems a natural and hashCode does cache the calculated value. The reason why fast code is indeed faster is because the compiler interns String literals <a href="http://java.sun.com/j2se/1.4.2/docs/api/java/lang/String.html#intern%28">http://java.sun.com/j2se/1.4.2/docs/api/java/la...</a>) <br>As to the static HashMap in the symbol class it is I am afraid a memory leak being a cache with no eviction strategy and one that can not be flushed, as a bonus getSymbol is not Thread safe! Static is indeed evil <a href="http://gbracha.blogspot.com/2008/02/cutting-out-static.html">http://gbracha.blogspot.com/2008/02/cutting-out...</a> Well to begin with String being mostly an immutable object computing the hashcode once seems a natural and hashCode does cache the calculated value. The reason why fast code is indeed faster is because the compiler interns String literals http://java.sun.com/j2se/1.4.2/docs/api/java/la…)
As to the static HashMap in the symbol class it is I am afraid a memory leak being a cache with no eviction strategy and one that can not be flushed, as a bonus getSymbol is not Thread safe! Static is indeed evil http://gbracha.blogspot.com/2008/02/cutting-out…

]]> By: Dave http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-74 Dave Thu, 17 Apr 2008 18:53:37 +0000 http://blog.dhananjaynene.com/archives/25#comment-74 Doesn't String.intern() effectively do what you want? Of course, this still seems like a pointless optimization. In my test of doing a million puts/gets, the difference in your two approaches about 100ms, or .104 milliseconds per operation. Not sure that would make a difference in most application contexts.... Doesn’t String.intern() effectively do what you want?

Of course, this still seems like a pointless optimization. In my test of doing a million puts/gets, the difference in your two approaches about 100ms, or .104 milliseconds per operation. Not sure that would make a difference in most application contexts….

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-73 Dhananjay Nene Thu, 17 Apr 2008 18:50:29 +0000 http://blog.dhananjaynene.com/archives/25#comment-73 @Jacob The *another* hash lookup you mention will try to check for the identity and then equality of keys. The class Symbol is constructed such that underlying strings are always identical (even if the symbol is constructed twice using two non-identical but equal strings). This is what makes the big difference. @Jacob The *another* hash lookup you mention will try to check for the identity and then equality of keys. The class Symbol is constructed such that underlying strings are always identical (even if the symbol is constructed twice using two non-identical but equal strings). This is what makes the big difference.

]]> By: Jacob Hookom http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-72 Jacob Hookom Thu, 17 Apr 2008 18:37:16 +0000 http://blog.dhananjaynene.com/archives/25#comment-72 How could that be faster when you are just deferring the regular String lookup to the Symbol map and then only slightly optimizing *another* hash lookup. If you make a case to keep a hard reference to the Symbol object outside of the map, then you might as well do that for the String itself and get the same optimization. How could that be faster when you are just deferring the regular String lookup to the Symbol map and then only slightly optimizing *another* hash lookup. If you make a case to keep a hard reference to the Symbol object outside of the map, then you might as well do that for the String itself and get the same optimization.

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-399 Dhananjay Nene Thu, 17 Apr 2008 18:26:00 +0000 http://blog.dhananjaynene.com/archives/25#comment-399 @Dave. To correct myself, I think if String.intern() is called each time a string key is used, one should get similar results without needing a Symbol class. @Dave. To correct myself, I think if String.intern() is called each time a string key is used, one should get similar results without needing a Symbol class.

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-398 Dhananjay Nene Thu, 17 Apr 2008 18:04:01 +0000 http://blog.dhananjaynene.com/archives/25#comment-398 @Dave. Have updated the post with the sources to be able to verify my findings. I missed the Strings.intern() part. I think it will certainly make for a simpler Symbol implementation. However the Symbol implementation will still be required. @Dave. Have updated the post with the sources to be able to verify my findings. I missed the Strings.intern() part. I think it will certainly make for a simpler Symbol implementation. However the Symbol implementation will still be required.

]]> By: Dhananjay Nene http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-397 Dhananjay Nene Thu, 17 Apr 2008 17:50:29 +0000 http://blog.dhananjaynene.com/archives/25#comment-397 @Jacob The *another* hash lookup you mention will try to check for the identity and then equality of keys. The class Symbol is constructed such that underlying strings are always identical (even if the symbol is constructed twice using two non-identical but equal strings). This is what makes the big difference. @Jacob The *another* hash lookup you mention will try to check for the identity and then equality of keys. The class Symbol is constructed such that underlying strings are always identical (even if the symbol is constructed twice using two non-identical but equal strings). This is what makes the big difference.

]]> By: Jacob Hookom http://blog.dhananjaynene.com/2008/04/turbocharge-your-string-keyed-hashmaps/comment-page-1/#comment-396 Jacob Hookom Thu, 17 Apr 2008 17:37:16 +0000 http://blog.dhananjaynene.com/archives/25#comment-396 How could that be faster when you are just deferring the regular String lookup to the Symbol map and then only slightly optimizing *another* hash lookup. If you make a case to keep a hard reference to the Symbol object outside of the map, then you might as well do that for the String itself and get the same optimization. How could that be faster when you are just deferring the regular String lookup to the Symbol map and then only slightly optimizing *another* hash lookup. If you make a case to keep a hard reference to the Symbol object outside of the map, then you might as well do that for the String itself and get the same optimization.

]]>