Does anyone have a working integration with karma-typescript they can share for an app fresh out of the CLI?
I’ve seen this boilerplate from @zewa666 but it hasn’t been updated in a while and the config is quite a ways from the latest CLI-generated project.
@mcorven
Is there a particular reason you are looking into Jasmine/Karma?
I’m asking because Jest offers it all in 1 convenient testing framework and there’s less configuration (again because it’s all in 1, there’s even code coverage built in). So if you do decide to go with it, you can take a look at Aurelia-i18n lib which is tested with Jest.
Our existing projects are largely based on the CLI generator (still very similar anyway). They’re using Karma+Jasmine. Moving to Jest isn’t an option at this time. I’m looking for ways to reduce the cold-start time for Karma.
My favorite testing setup at the moment is karma + mocha + webpack, which we’re using at vNext for about 12 months now.
With chrome headless launcher, it starts up really fast - the only thing that takes time is typescript compilation.
The setup is not trivial. That is an inherent problem with using an integrated stack rather than a single product, as @ghiscoding eluded to.
But, the time investment has been well worth it. Now we’ve got something that’s fast, reliable, easy to debug, and forces us to do things right (e.g. cleaning up after doing stuff with globals).
Jest has two major problems that make it a definite no-no for us going forward into the future.
It spins up a new environment for each and every test. You get parallelism but you lose, say, an order of magnitude (or two) in raw test performance. So overall your tests are still going to take anywhere from 10x to 50x as long depending on how granular the test code it. jest is the slowest testing framework out there
Apart from that, based solely on the fact that it runs in node with jsdom also makes any test using html / DOM inherently 2-3x slower than the browser (and never supports 100% of the features).
jest breaks node globals, for example it changes the identity of the Object function which causes any code relying on instanceof Object to fail (that means certain aurelia tests using the observer locator are going to fail where they should be passing).
It is easy to get started, something Jest shares in common with its nephew React, but what they also have in common is that once you realize it’s not going to scale, it’s often too late to turn back.
Aforementioned karma setup is not perfect, nothing is. But jest… they really take bad stuff to a new level. And that can be a hard thing to grasp when you’re already so invested in something!
Those 50k tests in vNext that take 3 minutes to run in chrome with karma, would easily take more than an hour with jest.
But still, 70% of those 3 minutes is spent in the testing framework itself. So we are looking into making our own lightweight test runner for vNext. To speed up our massive amount of tests, and also to give our users that “full package deal” kind of thing, and hopefully we neither need to deal with jest nor karma anymore.
Until then (and it may take a while), I’m 100% behind karma
Thanks @fkleuver I appreciate all the thoughts. I’ll start looking into the vNext testing config/structure and see what I can gain.
Jest is super easy to get started and for a relatively minimal solution like the Store and i18n plugin okayish when it comes to overall speed. In the process of moving the libs to vNext though we’re most likely going to switch them out again back to Karma or our own implementation because of the mentioned sideeffects.
Keep in mind though this doesnt mean you should immediately rewrite your current Setup. If you’re fine with jest now and have a relatively minimum set of tests, keep it as it is. If you’re planning a new project though, definitely take a good thinking
Just to add some numbers and more details why jest is less than ideal for certain Projects, especially for larger electron apps out of personal experience.
@mcorven Keep in mind the vNext test setup is probably more intricate and complex than you would need it to be, since it’s a mono repo and we put it together in such a way that it can run in multiple browser, conditionally report coverage, etc. But it does “contain everything” (except for typescript compilation, we actually do this in a separate step lately). Here’s the file you’d be looking for, it might be of some help but let me know if you have any specific questions
https://github.com/aurelia/aurelia/blob/master/packages/tests/karma.conf.js
You can easily swap out mocha for jasmine AFAIK
That’s interesting, I always read everywhere that Jest was 2-3x faster than Jasmine, so I assumed wrongly that it was the fastest JS testing framework lol. In my case, my Aurelia-Slickgrid lib which has now ~1400 tests is averaging out to 1.45 sec on CircleCI (however there’s no DOM integration tests yet). In my case, I’ll definitely stick with Jest as it’s easier to setup and I’m getting more familiar with it, but it’s definitely good to know and hear about all these other side effect and comparison.
@fkleuver
I guess the other advantage you have with “karma + mocha + webpack” setup is that Cypress (E2E) is also written with mocha.
@ghiscoding The majority of the world also believes that React is the fastest SPA framework out there when it is in fact one of the slowest. Facebook certainly knows how to market their stuff.
That said, I haven’t used Jasmine in a while now. It’s possible that jasmine’s assertions are just a bit slow, many assertion and mocking libraries are. This is why I ported node’s assert framework over in a way that it can also run in the browser and our vNext tests now run about 4x faster (we do a lot of assertions per test).
To give you some concrete numbers:
So plain JS tests are always going to be faster in nodejs runners due to the lack of console capturing / websocket overhead. DOM tests are always going to be faster in the browser because native HTML parsing/rendering obviously beats a JS implementation (jsdom)
All of these tests are single-threaded. Running them over multiple threads, like jest does, would drastically cut the time even further, but there may be other overhead for making the tests “multi-threadable” in the first place. Jest performance is certainly suffering from their choices in that regard.
That said, performance itself shouldn’t be a reason to pick one testing framework over the other for the majority of users. If jest helps you get up and running quickly and reduces your time-to-market, you’re doing good. This is only an advantage though if you don’t already have a working setup from another project which you can copy-paste.
For normal end user apps I wouldn’t necessarily recommend integration tests for the frontend in the first place. Unit test your normal vanilla components and e2e test the interactions, and you’ll be pretty good. For a framework that’s serious about testing, it’s another story.
But when you happen to run into any problems caused by jest breaking some globals, then you’re in a very hard place, and that’s my primary motivation for recommending against it - no one cares if their tests take 2 or 4 seconds to run.