It all started with my attendance at Brian Sam-Bodden’s presentation at Spring IO 2022, which raised a great curiosity about how powerful Redis could be as a primary database. The main reason was speed. A writing operation takes less than a millisecond to be performed due to Redis data being stored in memory. I needed to test this. And since I’m a Kotlin and Spring utilizer. Why not use Redis OM Spring, the library that Brian was advocating for in his presentation and to which he is the main contributor? What I didn’t know was that this curiosity would lead me to my first open-source contribution. A contribution that helped Redis OM Spring persist data 10 times faster.
- Introduction (How I decided and got the chance to contribute)
- Getting my hands dirty
- Understanding Spring Data Redis (taking a step back)
- Understanding Redis OM Spring (diving deeper)
- Overwriting the saveAll method
IntroductionMy first tests were a bit frustrating, the speed for inserting 10 thousand records was of 35 seconds, not mind-blowing at all. So I tweeted: https://twitter.com/raphaeldelio/status/1544781772776423430?s=61&t=lj_QjBsploDve48wDphaVg This tweet started a thread in which Brian Sam-Bodden and Steve Lorello share that it’s probably a matter of pipelining. Besides that, Brian said they were first tackling usability, and then they would tackle performance. That ignited my curiosity even further. I was just getting started with Redis and I don’t even know what pipelining is. So I decided to forget about it for a second and get started with Redis Server, Redis CLI, and Redis Insight first. Redis OM Spring does a great job at usability. The abstraction is so well done that I was able to get started and start using Redis even before understanding how Redis works. However, understanding the basics of Redis was more important at this stage. And so I did it. I learned a few things, I got started with running Redis locally, then learning the basics of Redis CLI, Redis Insight, how it persists data to the disk, pipelining, transactions, and a few other things. Still, a long path to go, but enough to get started with pipelining on the Kotlin side.
The architecture of Redis OM SpringRedis OM Spring is built on top of Spring Data Redis, which in turn, is built on top of Jedis or Lettuce, two different drivers for connecting Java to a Redis Server. Lettuce seems to be more powerful as it allows asynchronous calls and clusterization, which I haven’t dived into yet. Jedis allows commands to be pipelined, everything I needed. I decided to go with Jedis. Jedis is very easy to use, you send commands as you would be sending to the server as well. For example, to set a simple String key, all you need to do is:
And to pipeline commands, you can do it as:
Cool. Let me see how long it takes to add 1 million simple String keys… Now, that’s mind-blowing… 4 seconds? So I tweeted again: https://twitter.com/raphaeldelio/status/1563232997930766336?s=61&t=AWln5xRDk-Vu_PsduGu_AA And sent a message to Brian if there was any way I could actually contribute to the project: Brian asked me to do a comparison with Spring Data Redis as well. His intention was to understand if this was something Redis OM Spring had inherited from. These were the results I shared with him and in my GitHub repository (Check it here) as well: With this exercise, I also took the opportunity to dive deeper into Spring Data Redis and Redis OM Spring implementations, which allowed me to engage in a conversation on where to get started with Brian. Brian directed me to the right places in the code to refactor: The problem was that Redis OM Spring was inheriting the saveAll methods from Spring Data Redis. And Spring Data Redis was doing one insert at a time, not benefitting from the pipelining feature. Redis OM Spring allows us to manipulate both Hashes and Documents. Therefore, we had to overwrite this method in two different places: SimpleRedisEnhancedRepository and SimpleRedisDocumentRepository.
Pipeline pipeline = jedis.pipelined(); pipeline.set("key1", "value1"); pipeline.set("key2", "value2"); pipeline.sync();