Hmm, I didn’t test this many tests. But you can run all tests in parallel — two at a time — and we should be good. The limitation isn’t the framework; it’s your budget and the tier you have on OpenAI. If you have Tier 2, which allows more requests to the OpenAI API, you could potentially run 1,000 tests in a single run.
Basically, it’s built on top of Playwright — kind of like Selenium. The pros are that it’s easy to write tests in English and you don’t need to maintain them. The cons are that you spend tokens, so it costs money for every test run.
Unfortunately, yes — you need to spend money for every test run. But you’re paying either way. Let me explain: I work at a big company with a large QA team. People write tests in JS/TS/Python. It takes time to write them, and then more time to maintain them when they break due to UI changes. Our team spends a lot of time fixing flaky and broken tests. Time is money. So leadership has to make a decision: should people spend less time writing and maintaining tests and focus more on quality, even if it means paying for test runs?
How good is it in terms of scalability? Im looking for something that wont crap out when we get to 100s or 1000s of tests
Hmm, I didn’t test this many tests. But you can run all tests in parallel — two at a time — and we should be good. The limitation isn’t the framework; it’s your budget and the tier you have on OpenAI. If you have Tier 2, which allows more requests to the OpenAI API, you could potentially run 1,000 tests in a single run.
What other LLMs do you support?
Only Open AI right now. Maybe i will add Ollama or Gemini in future
So it potentially can outperform/ substitute selenium for frontend testing?
Basically, it’s built on top of Playwright — kind of like Selenium. The pros are that it’s easy to write tests in English and you don’t need to maintain them. The cons are that you spend tokens, so it costs money for every test run.
So, are you saying that I should spend money on every test run?
Unfortunately, yes — you need to spend money for every test run. But you’re paying either way. Let me explain: I work at a big company with a large QA team. People write tests in JS/TS/Python. It takes time to write them, and then more time to maintain them when they break due to UI changes. Our team spends a lot of time fixing flaky and broken tests. Time is money. So leadership has to make a decision: should people spend less time writing and maintaining tests and focus more on quality, even if it means paying for test runs?
Love it. Very easy to setup and start adding e2e tests using prompt.
Thank you! It’s free and open source. Contributions are welcome! I’ll be releasing a new feature soon that will make it even easier.
This stuff is pretty helpful, love to test with it!
I have a 5-minute quick start guide on the website. It’s easy to get started — you just need an OpenAI key
Can I use Olama?
Unfortunately, no — not for now. Maybe in the future
Do you support Gemini?
No, maybe in the future
Great stuff!