Sunday, June 4, 2023
HomeTechnologyLichess got a big didn't go as planned

Lichess got a big didn't go as planned

Lichess gets a major upgrade. It didn’t go as planned.

Lichess is a 100% open source/freeware chess website used by millions of players to play billions of games. I decided to write it in scala many years ago and never looked back. It has features that are important to me:

    Type safety: Compiler as co-pilot Functional Programming: Functions as Building Blocks Performance and Ecosystem: JVM as a Solid Foundation

  • into Scala 3
    Lichess is Built on Scala 2, I was very excited to upgrade when Scala 3 was released last year.

 However, I chose to wait a full year for the language to stabilize and the library ecosystem to catch up.   Last month, I decided that the wait was over and it was time for a massive Lichess upgrade. Can I wait longer? Of course it does, but I don't see why, and I'm honestly craving the latest features.   

What's new in Scala 3 This is not so much an evolution of the language as a complete overhaul because the compiler is Rewritten from scratch . However, compatibility is preserved as much as possible, thereby simplifying migration. Here are some of my favorite Scala 3 features:

Opaque types ) stronger typing and zero runtime cost, what's not to like? Strings (such as user IDs) and other primitives can be replaced with appropriate types understood by the compiler.

opaque type UserId=String def find (id: UserId) // This function does not accept any string, only UserId value

I actually found and fixed some old obscure bugs caused by using strings, while changing them to


Clearer syntax Prominent indentation and optional curly braces make our code look like python, which is cute. Luckily, the comparison ends there ;)
 object chess: def turnColor(ply: Int)=if ply % 2==1 then White else Black  

Improved type inference

We want types, not boilerplate. Sometimes it's best to let the compiler figure out what something is. During the Scala 3 migration, I've been able to remove a lot of type annotations, which makes me love typing.

Better Context Abstraction

implicit has been replaced with the private keyword use, given and extensions .

It makes the code easier to understand because of its intent than using generics implicit keyword.

Nice looking enum
 more code Sugar stuff, but I always prefer simplicity and expressiveness : 

 enumeration DrawReason: case MutualAgreement, FiftyMoves, ThreefoldRepetition, InsufficientMaterial  It doesn't look like much, but it comes with batteries.   

New Export


it works like import, but expose the function and value. It makes the composition more concise. / / Before: def rating=glicko.rating def deviation=glicko.deviation // after: export glicko.{ rating, deviation } If better composition means less inheritance, count me in.

New Inline keyword

 While the previous @inline annotation was a best effort thing, the new 

inline keywords are guaranteed to be available during compilation Inline.

 It is a powerful tool that should be used with care.   

 The list could go on and on; Scala 3 is much more than that! This post is already too long, so I'll cut down on the followers.  

Lila is a large program that handles 2000 HTTP requests per second, downloading 5000 moves of chess while doing a lot of other things I'd rather not list here. So, yes, migrating it was horrible, and I fully expected some kind of disaster. Let's see how it goes.

 Fortunately, metals and bloop can be nice Handles Scala 3 as it should, which provides full language support for my code editor. It's a very comfortable experience.   All we need now is some brave souls to improve Scala 3 support for treesitter so we can enjoy proper code coloring of the new language syntax.   

update my code
 this is simple and fun part, especially since the compiler does most of the work for me. I'm actually rewriting more code than before because I can't resist converting some  implicit 
given here and there, and use opaque type.   At some point I had to rewrite the Glicko2 rating system from Java to Scala 3 because the compiler complained about Java in my project. No one noticed the drop in ratings, so I guess it worked.   

3rd party library This is where things get a bit furry place. Lila is built on Play Framework which has not been ported to Scala 3 yet. So I forked it and deleted it to remove everything we don't need - which is actually most of the framework. Migrating to Scala 3 became very easy once Play was reduced to a few small libraries (HTTP/netty server, routing and forms). Most other dependencies, such as the MongoDB driver, template engine, or the lovely function cats, have been upgraded to Scala 3. As for libraries from the Java ecosystem, such as our redis driver, they work as usual.

into production
 when everything compiles,  I shipped it. To everyone's surprise, apart from a few bugs I made while rewriting thousands of lines of worked. It just does. No explosions, no cryptic bugs, no memory leaks, no performance hits. This is quite unexpected.   With Scala 3 in production, I'm free to rewrite the code more deeply to gradually take advantage of Scala 3 features.   

JVM tuning until one morning, Instead of deploying the previous day's changes, I let it run for an additional 24 hours. We then saw JVM CPU usage rise to alarmingly high levels and anomalous patterns emerged. No obvious culprit in the thread dump... I couldn't figure it out and ended up asking for help - read all about it in my previous blog post. The Avengers rallied and saved Lichess: just the JVM needed some tweaking. The HotSpot compiler was running out of code cache, and once we gave it some more, things suddenly got a lot better.

Incredibly fast Results, Lichess Now faster than ever. Running previous versions of Scala 2 was also limited by lack of JVM tuning. The effect is less spectacular, but still: we basically run Lichess with the parking brake pulled. Lichess now runs on Scala 3 without a parking brake and is faster. To be able to tell if Scala 3 itself is faster, we have to roll back to Scala 2 and try it with appropriate JVM tuning. I don't want to do that, sorry! Once you try Scala 3, you can never go back.

Final Words
Ever since Scala 3 was announced, I've been both terrified and terrified of this migration excited. Given the size of our codebase, I expected a disaster, but all we got were a few bumps in the road. We're smooth sailing now, all modern and future proof, and it feels kind of awesome. It took only a month to fully migrate Lichess, from the first code change to confirming that it was running flawlessly in production (current uptime: 7 days). For every refactoring or migration, success and speed depend heavily on static typing and compiler quality.

10/10 will migrate again

Thank you
Big thanks to the Scala 3 team for their incredible work on this new version of the language. Changes are also well documented. I am also eternally grateful to all the people who worked on the Scala tools, to the wonderful community of developers who helped me, guided me, and sometimes even wrote complex code directly for me when I needed it. Thanks to all Lichess players, and to everyone who supports this beautiful project with donations!



Please enter your comment!
Please enter your name here


Featured NEWS