ScalabilityTestResults

This is a rough rendering of a page from the old Prevayler wiki. Please see the new wiki for current documentation.

Many people dismiss Prevayler's scalability test results as a hoax. @:)

(To understand how they are possible, see PrevalenceSkepticalFAQ.)


Query ScalabilityTest Results: (queries per second)                      
                      
{#                            
Prevayler | Oracle | MySQL | SQLServer | CombinedProcessorSpeed | More Details                          
62880.46 | 9.15 | | | 400 MHz | ScalabilityTestByEduardoAkatsu                           
144205.91 | | 28.08 | | 500 MHz | ScalabilityTestByEduardoSimioni                           
215543.43 | | 75.87 | 95.46 | 900 MHz | ScalabilityTestByDanielSantos      
#}                            
                  
                
Query Scalability Ranking: (based on results above)                  
                  
{#                  
Rank | System | Average QueriesPerSecondPerMHz | Comparison to Prevayler                  
1 | Prevayler | 228.369 |               
2 | SQLServer | 0.10607 | 2153 times slower than Prevayler           
3 | MySQL | 0.07023 | 3251 times slower than Prevayler           
4 | Oracle | 0.022875 | 9983 times slower than Prevayler           
#}               
                    
----            
Manipulation ScalabilityTest Results: (manipulations per second)                      
         
{#                            
Prevayler | Oracle | MySQL | SQLServer | CombinedProcessorSpeed | More Details                          
106.53 | 15.50 | | | 400 MHz | ScalabilityTestByEduardoAkatsu                           
62.29 | | 115.44 | | 500 MHz | ScalabilityTestByEduardoSimioni                           
55.42 | | 298.31 | 144.50 | 900 MHz | ScalabilityTestByDanielSantos                           
#}                            
                  
                
Manipulation Scalability Ranking: (based on results above)                  
                  
{#                  
Rank | System | Average ManipulationsPerSecondPerMHz | Comparison to Prevayler                  
1 | MySQL | 0.28 | 1.87 times faster than Prevayler        
2 | SQLServer | 0.16 | 1.06 times faster than Prevayler        
3 | Prevayler | 0.15 |        
4 | Oracle | 0.039 | 3.89 times slower than Prevayler        
#}               
            
            
----            
Hard to believe?                               
You can easily run the ScalabilityTest for yourself. Just Download the latest Prevayler alpha release and read the readme.txt file included.   
                    
We can certainly use more results. Please send your own results to prevayler-scalability@lists.sourceforge.net and they will be published on this page. Running the tests for other DBMSs is a good idea too.
   
Thanks,   
The PrevaylerTeam.                                 
                    
----                                  
See: ScalabilityTest.
I'd like to see comparisons on startup time between DBMS based applications and Prevayler based apps.  Obviously, this isn't going to be your strong point, but it would help complete the picture, show how much you pay for the improvements mentioned above...

Sure. According to DouglasGiacomini, Prevayler recovers 300.000 simple object creation commands in 50 seconds on his home PC (IDE hard-drive). I shall produce better benchmarks, though. If you have more results, please publish them here too. -- KlausWuestefeld.


What about the impact of maintaining multiple indexes in a given entity? Obviously, for hierarchical storage and simple hash lookups, objects in memory are superior to databases. But when replacing a database, could you be losing some of the effeciencies of heavily optimized DB indexing techniques? In other words, if you use prevayler, do you have to become an indexing expert in order to balance memory usage for multiple indexes against performance considerations?

Hardly. Performance is a non-issue. In the WORST CASE, indexing can be done exactly the same way your database does it @;) but without inter-process comunication and without object marshalling/unmarshalling. We are talking 3 to 4 orders of magnitude faster.
The real issue is the complexity of your code. You can write indexing code yourself or you can use a library for that. See: HowDoIQueryMyObjects? You are no longer restricted to indexing schemes that must work well with disk blocks, such as the database indexing schemes. --KlausWuestefeld


Can I suggest that without detailed configuration details of the box, these figures still don't mean that much to me. What's the memory size & speed?  What sort of boxes are these running on?  What's the IO subsystem like?  It's all well & good to test these on my PC at home, but that hardly compares to the hardware I'd deploy to.  That's where it really counts after all.

On every comparison, the same machine was used for Prevayler and the database in question. Feel free, though, to run the tests on any hardware you want and publish your results here. --KlausWuestefeld.


And what's that you say? In process operations are orders of magnitude faster than calling out to another system.  WOW ;)  You talk elsewhere about still being able to remote query using RMI, CORBA, JSP's perhaps...  Can we see some performance comparisons based on these, where we're no longer talking about everything working in the same VM?

The client-to-appserver connection will be equally slow with Prevayler or with a database. The appserver itself, though, is 3 to 4 orders of magnitude faster with Prevayler because, as you well spotted, you can have a cohesive ObjectServer in a single process. With the database/appserver architecture you cannot. That's precisely the point. --KlausWuestefeld.


I need fault tolerance.  Load balancing.  Multiple VM's accessing the same data at once.  How does it compare under these conditions?

Well, if you manage to put 9983 machines in an Oracle cluster (with zero overhead, mind you), then you might get the same performance scalability as Prevayler on a single machine. The best thing is we don't even have to discuss. Just run the tests. --KlausWuestefeld.

What would be interesting is to test against an in-memory Java DB like McKoi ;-) --StephanSchmidt

Just do it. The test will run on anything that has a JDBC driver. --KlausWuestefeld


Yes, if I have some time left. But I guess we will change either way, because right now I think McKoi does heavy caching and SnipSnap does heavy caching resulting in keeping the data twice in memory. --StephanSchmidt


"What are you waiting for? You're faster than this. Don't think you are, know you are..." --Morpheus in The Matrix



Are you not comparing a Database that is using a remote protocol (JDBC in this case) to Prevayler which is all in one VM?

Yes. This question was already asked and answered above. --KlausWuestefeld

Seems to me that to get an Apples/Apples comparison your benchmark should have a remote client (using RMI, or something like that).

No. Feel free to run your database in the same VM as the test. @;) --KlausWuestefeld

As a follow up to the above question - please don't dismiss the remote issue as "we dont need to do it that way". If this technology is going be accepted as a viable persistence mechanism, it is going to need to deal with multiple remote clients.

No. Persistence mechanisms do not have to "deal with" remote access. That would be coupling. They just have to "allow for" remote access. That is what "cohesion" means. --KlausWuestefeld.

Security in particular is a concern. Having a remote protocol makes it much easier to control access to the data.

Doing access-control directly on data is brutally breaking encapsulation but, anyway, you are finally free to use whatever remote protocol you want. --KlausWuestefeld

See: NoMorePorridge.



Any non trivial application is going to have to use a remote protocol of some description. For example, if I am Bank - I am going to want some degree of isolation between my core transactional data and my application clients. RDBMS provide some measure of isolation. Yes, I am free to use any remote protocol I want - but I have to use something - and your benchmark does not factor this in. Once again - I think your benchmark is not comparing apples to apples.
<p>
Don't get me wrong - I really like the idea of prevayler! Writing enterprise apps is very difficult - and a large part of that is dealing with impedance mismatch between the object and relational worlds. That being said, the issues of security, isolation, etc. are going to have to be dealt with if this technology is going to be usefull.
<p>
One approach might be to wrap Prevayler in a distributed technology like Jini (using RIO, etc.). A persistent capable application could expose business methods (e.g. Account.updateBalance, etc.). You deploy a bundle of Biz Logic + Persistence to the persistence service. Remote clients can discover the service (via the lookup service) - and you get to leverage all the other Jini goodies (access control, robust service recovery, etc).  Now that would be cool! I think there is a lot of potential synergy here. For example - your replication services could automagically discover each other. A deployed application could use the Quality of Service features of RIO to ask for the level of replication they desire.
<p>
-- Warren  mailto:wstrange@telusplanet.net


Simply: I AM comparing apples to apples. You are wrong to state otherwise and that is not your point, anyway. Your point is: YOU ARE NOT INTERESTED in apples. You are interested in REMOTE APPLES. Regarding "remote apples" then, here we go again: The client-to-appserver connection will be equally slow with Prevayler or with a database. The appserver itself, though, is 3 to 4 orders of magnitude faster with Prevayler because, as you well spotted, you can have a cohesive Object Server in a single process. With the database/appserver architecture you cannot. That's precisely the point. I like the Jini idea. --KlausWuestefeld


I think your claim of having an appserver that's 3 to 4 orders of magnitude faster with Prevayler is valid for the case where a remote client (if it's an appserver, there are going to be remote clients) is making a single request that would result in a very complex query over several objects (thousands, it looks like).  The performance of a Prevayler-based appserver vs. a RDBMS-based appserver would look different for the case of many remote remote clients resulting in a few queries each. I think Prevayler would still be faster, but probably not 3 or 4 orders of magnitude faster. --Mark Kinzie

Data drilling and mining activities as well as huge batch processes, where no client interaction is involved, also benefit from the 3 to 4 orders of magnitude acceleration. But regarding small queries from remote clients you are certainly right. The server will still have to process network packets and format responses for the clients. Using Prevayler, the time to execute the business query itself is negligible. Prevayler actually raises the performance bar for all other components in your architecture. --KlausWuestefeld



I wonder if anyone would be prepared to generate some benchmarks for a Berkely DB based solution or similar. I would if I had the opportunity but I don't.