To increase the performance of your model, you should optimize for the common case, even if it means that some uncommon cases become much slower. Also, it is usually more important to reduce how often the device model is invoked, than reducing the time spent in each invocation.
The simulator framework can help you avoid unnecessary model invocations. For example, rather than having the device model poll a memory location repeatedly, you can set a breakpoint which is triggered only when the value at that location changes.