Groovy @EqualsAndHashCode is slow!

Found this out benchmarking an app at work. We’re pulling a fair amount of data from the database via hibernate, and hydrating those objects into a set of domain objects. It’s a pretty straight-forward set of domain objects, involving stores, ownership information, operation information (store hours, addresses, etc). The pulling of data was terribly slow, though.

Putting the app through it’s paces in YourKit, however, revealed something we didn’t expect: the hashcode generation for a number of our domain objects was dominating the list of hotspots. In particular, NullObject#is.call(….) was taking an inordinate amount of our cpu time.

Now, our data can have a good number of null values; not every field is populated … and that appears to be the crux of the problem.

I took one of the prime culprits from our domain and wrote a quick unit test just to run over a large number of objects and generate hashcodes, and output how long it takes. I also generated a set of hashcode/equals method via Eclipse … and hand-generated the same same via Apache’s EqualsBuilder and HashCodeBuilder, as well a Guava’s Objects utility (and for good measure, Java 7’s equivalent Objects utility).

What I’m seeing in 10M hashcode generations is roughly like this:

   Groovy Time:  44001 ms
   Eclipse Time:  4488 ms
   Apache Time:   4361 ms
   Guava Time:    5912 ms
   7 Time:        5919 ms

Clearly, if you’re going to have null attributes, then it’s worth the time to produce your own equals and hashcode methods; with the clean appearance of the Apache utility methods, I’m definitely leaning that direction.

 

Advertisements
Posted in Software | Leave a comment

Groovy 2.4.x bug

This is a short blog post; just read the details here, though the bug actually is more generic than the initial bug report suggests (generics are not required).

There is a bug in the 2.4.x groovy where an incorrect super method invocation can occur. We discovered this while looking at making a move from 2.3 to 2.4, but now we’re hesitant because at least one of our projects was coded this way. We don’t know if any 3rd party libraries we’re using are similarly coded.

Basically, if you have a call hierarchy C invokes super on B, which invokes super on A, but C doesn’t actually override the method it’s invoking via super … you’ll not get the B method (as you would in Java), but you’ll get the A method.

If you’re reading this, please wander over the apache jira site and vote this issue up; it definitely needs worked.

UPDATE: Very quick work by Jochen Theodorou — the bug is fixed, and will be out with the Groovy 2.4.8 release (and is also fixed on the 2.5.0-beta-1 branch).

Posted in Software | Leave a comment

Locked out of Android Phone

Okay … it’s been a terrible week for phones for me. I went jogging last week, with my phone in an exercise carrier strapped to my forearm. When I got home … the phone was pretty sweaty. Having dried it off, I did open it to check it out and … yeah, sweat inside the case, too (LG G4 btw). After that, all the sensors seemed to be dead (rotation didn’t work, proximity sensor was dead, etc. … plus random shut downs). A $20 warranty replacement is forthcoming from T-Mobile, so … all should be good.

Today, my wife and I took my dogs on a walk, and when I got back to the house and pulled the phone out of my pocket, apparently I had quite a number of incorrect passwords entered into the phone. I got a nice message about verifying my identity by logging in with my google credentials. Hey, no problem! I do so and … doesn’t work. Do some digging — and it turns out you have to use the original google account and original password that you set the phone up with. Alright … I did that just 3 months ago (warranty replacement — told you it was bad) so the user id and password should have been correct.

Nothing I tried would work; I starting thinking a factory reset was in the cards for me. Then I stumbled across this forum post, in particular a post by wgrotheer. As it turns out, for security, I had enabled 2-step verification. You MUST turn 2-step verification off in order to authenticate to your phone. So … quick run out to Google, turn it off, and viola! I’m logged back into my phone.

Thank you wgrotheer, whoever you are. Your post back in 2011 saved me!

Posted in Uncategorized | Leave a comment

BitBucket — who has access to what?

This blog post was a god send: http://www.stevenmaude.co.uk/posts/managing-a-bitbucket-users-permissions-when-theyve-left-your-team

This pointed us to an area of the admin section of bitbucket that provided us with exactly what we were looking for today. It’s not what permissions are still around when they left, but … given a user, what is the list of repositories to which they have access.

This provided exactly what we wanted, and it was very easy to get to once you know how (and, in my case, if you have admin rights to bitbucket … which I do not with this client).

Posted in Software | Leave a comment

Dynamically Clearing Tables While Testing

I had to blog about this, or I’m going to forget about it. Isn’t that always the case? We write some utility, and run into some issues, and then 6 months from now we run into it again, but so many other things have come and gone that we’ve forgotten about them.

This has to do with integration testing, and clearing the state of the database between runs. In particular, clearing the contents of database tables. We’re not so much concerned with rebuilding the database — just the data that may be contained within it.

Generally, in our situation, we use liquibase to set up our database, and we use h2 for a fast, in-memory database in a Spring JPA codebase. Between tests, however, it can be a pain to always keep in mind what tables have data written to it, what foreign-key relationships need to be considered when deleting objects, etc. So why not have a utility class that just obtains the set of tables, ignores the FK constraints, and clears all the tables (except for liquibase).

Here’s what we came up with:

package com.sonicdrivein.drivein.services.util

import javax.persistence.EntityManager

import org.springframework.orm.jpa.JpaTransactionManager
import org.springframework.transaction.TransactionStatus
import org.springframework.transaction.support.DefaultTransactionDefinition

/**
  * Class containing test helper methods.
  */
 class DaoHelper {
 
     /**
      * Method to clear out all tables so that tests can be
      * run independently.
      */
     public static void clearTables(EntityManager em) {
 
         // Dynamically get all tables available to the dao and delete their contents, one by one.
         def schemas = getSchemas(em)
         schemas.each { schema ->
             def tables = getTables(em, schema)
             tables.each { table ->
                 if (table != "show tables") {
                     table = table[0]
 
                     System.out.print("Clearing data for table: ${schema}.${table} ... ")
                     // Don't muck with liquibase
                     if (table != 'DATABASECHANGELOG' && 
                         table != 'DATABASECHANGELOGLOCK') {
                         em.createNativeQuery("alter table ${schema}.${table} set referential_integrity false").executeUpdate()
                         em.createNativeQuery("delete from ${schema}.${table}").executeUpdate()
                         em.createNativeQuery("alter table ${schema}.${table} set referential_integrity true NOCHECK").executeUpdate()
                         System.out.println("done.")
                     } else {
                         System.out.println("skipped.")
                     }
                 }
             }
         }
     }
    /**
     * For the cases (rare) where we need to start/end transactions manually, this will definitely help.
     * You'll likely have to @Autowire a JpaTransactionManager into your test code, and pass it to this method.
     *
     * @param jtm The JpaTransactionManager
     * @return TransactionStatus
     */
     static TransactionStatus startTransaction( JpaTransactionManager jtm ) {
          DefaultTransactionDefinition transDef = new DefaultTransactionDefinition()

          return jtm.getTransaction(transDef)
     }

    /**
     * For the cases (rare) where we need to start/end transactions manually, this will definitely help.
     * Commit the transaction for this jtm & txstatus.
     *
     * @param jtm
     * @param txstatus
     */
     static void commitTransaction( JpaTransactionManager jtm, TransactionStatus txstatus ) {
          jtm.commit(txstatus)
     }

    /**
     * For the cases (rare) where we need to start/end transactions manually, this will definitely help.
     * In the context of this jtm, start a transaction, clear the tables, and then commit the transaction.
     *
     * @param jtm
     */
     static void clearTables( JpaTransactionManager jtm ) {
          TransactionStatus txstatus = startTransaction( jtm )
          EntityManager entityManager = jtm.entityManagerFactory.createEntityManager()
          entityManager.joinTransaction()

          clearTables(entityManager)

          commitTransaction(jtm, txstatus)
     }

    /**
     * For the given dao, return all existing schemas.
     * @param dao
     * @return
     */
     private static def getSchemas(EntityManager em) {
          def ret = em.createNativeQuery("show schemas").getResultList()
          def schemas = []
          ret.each { schema ->
            if (schema != "show schemas" &&
                schema != "INFORMATION_SCHEMA") {
                   schemas << schema 
            }
          }
          return schemas
     }

     /**
      * For the given dao, and schema, return all existing tables. 
      * @param dao 
      * @param schema 
      * @return 
      */ 
     private static def getTables(EntityManager em, String schema) { 
          def ret = em.createNativeQuery("show tables from ${schema}").getResultList() 
          def tables = [] 
          ret.each { table ->
               if (table != "show tables for ${schema}") {
                    tables << table
               }
          }

          return tables
    }
}

The use of this is quite simple. For most of our integration tests, we simply autowire in the EntityManager via @PersistenceContext:

@PersistenceContextEntityManager entityManager

Then, in the @After block, just invoke the DaoHelper to clear all your tables:

@Aftervoid tearDown() {     DaoHelper.clearTables( entityManager )}

And voila! The tables are cleared, and your tests are once again independent.

It’s a little more complicated if you’re not having Spring do all your transaction management; this is why the additional methods involvinga JpaTransactionManager are there. We had a situation where we are using both Mule, and Spring-JPA. Our integration test required that we spin up Mule… but that, in a testing context, caused some problems with the Spring integration. Our solution was to have our test case extend a base class based on this post by Ivan Krizsan:

package com.mycompany.services.util
 
 import org.mule.api.registry.RegistrationException
 import org.mule.api.registry.Registry
 import org.mule.tck.junit4.FunctionalTestCase
 import org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor
  
 // Kudos go to http://www.ivankrizsan.se/2015/12/13/spring-dependency-injection-in-mule-functionaltestcase/
 
 /**
  * Mule functional test case that will perform Spring dependency injection on the test class instance
  * prior to executing tests.
  * For dependency injection to be performed, the Mule context must be started before the {@code doSetUp}
  * method is executed.
  * For details on dependency injection performed, please refer to {@code AutowiredAnnotationBeanPostProcessor}.
  *
  * @author Ivan Krizsan
  * @see AutowiredAnnotationBeanPostProcessor
  */
 public abstract class SpringDIFunctionalTestCase extends FunctionalTestCase {
  
     @Override
     protected void doSetUp() throws RegistrationException {
         if (muleContext.isStarted()) {
             final Registry theRegistry = muleContext.getRegistry()
             final AutowiredAnnotationBeanPostProcessor theAutowiredAnnotationBeanPostProcessor
             try {
                 theAutowiredAnnotationBeanPostProcessor =
                     theRegistry.lookupObject(AutowiredAnnotationBeanPostProcessor.class)
             } catch (final RegistrationException theException) {
                 logger.error("An error occurred retrieving AutowiredAnnotationBeanPostProcessor", theException)
                 throw theException
             }
  
             if (theAutowiredAnnotationBeanPostProcessor != null) {
                 theAutowiredAnnotationBeanPostProcessor.processInjection(this)
             } else {
                 logger.warn("No AutowiredAnnotationBeanPostProcessor in the Mule registry, "
                     + "could not perform dependency injection on the test instance.")
             }
         } else {
             logger.warn("Mule context is not started, no dependency injection on test instance performed.")
         }
     }
 }

Using this, coupled with the DaoHelper, we can have our Mule integration test perform some autowiring … but transaction management within the test couldn’t reliably be done with @Transactional. Hence, the DaoHelper methods we added. Now our integration tests can extend SpringDIFunctionalTestCase, autowire the JpaTransactionManager, and we can invoke the DaoHelper.clearTables( jpaTransMgr ) method. We can also manually start and commit transactions quite easily.

Posted in Uncategorized | Leave a comment

Spring jpa:repositories and subpackages

This one has been biting me for a few days; I just had to jot a few notes down about it.

Everywhere you look, the documentation about the <jpa:repositories> spring config element says that the base-package and all subpackages will be scanned for entities that are marked with @Repository (or one of its derived classes). What’s I’ve found is that this just isn’t true. We used:

<jpa:repositories base-package="com.mycompany.foo" .../>

and have a Spring-derived Dao in the com.mycompany.foo.bar package. We end up with an exception in our test case something to the tune of “not an managed type: java.lang.Object”. If, however, we replace the above spring snippet with:

<jpa:repositories base-package="com.mycompany.foo.bar" .../>

then everything is fine.

I wonder if everyone says it scans subpackages, but nobody actually uses it that way?

Note that if you have multiple packages, you can provide a comma-separated list of packages, or (so they say), you can provide a wild-carded base-package. Good luck!

Posted in Software | Tagged , , , , | Leave a comment

Hibernate 4 & the MSSQL Money type

This post covers a situation we ran into where Hibernate 4 wouldn’t map the MS SQL Server Money type to a BigDecimal in our groovy code. Hibernate 3, it turns out, does so just fine, but in our new app, with Hibernate 4 … it just wouldn’t.

The solution we hit on is a bit odd, but it works. First, we had been using a custom UnicodeSQLServerDialect like this:

class UnicodeSQLServerDialect extends SQLServerDialect {
    public UnicodeSQLServerDialect() {
        super()
        registerColumnType(Types.VARCHAR, 255, 'nvarchar($l)')
    }
}

We made an addition to the dialect in order to handle the money type, but then ran into an issue where the decimal type started being mapped as if it’s money. What we ended up with is this:

class UnicodeSQLServerDialect extends SQLServerDialect {
    public UnicodeSQLServerDialect() {
        super()
        registerColumnType(Types.VARCHAR, 255, 'nvarchar($l)')
        registerColumnType(Types.NUMERIC, 254, 'decimal($p, $s)')
        registerColumnType(Types.NUMERIC, 255, 'money')
    }
}

and some changes to our domain classes which are odd, but work. Namely, if an entry is a decimal in the db, we specify a length=254, and if it’s money, we specify a length=255.

    @Column(name = 'Quantity', length=254) // Map to decimal(?,?)
    BigDecimal quantity

    @Column(name = 'Price', length=255) // Maps to money
    BigDecimal price

It’s really strange, but digging through the hibernate code, it looks like the mapping of types relies heavily on the length.

Strange … confusing … but it works.

Posted in Software | Tagged , , | Leave a comment