Open Source Ab Testing Framework Image

Open Source Ab Testing Framework

Open Source Ab Testing Framework (OSATF) is an open source Ab testing framework that supports a wide variety of online advertisement units including banner ads. It has a pluggable architecture to support different advertisement publishers and provides a unified simple way for webmasters to build, manage and monitor their AB test campaigns.

Open Source Ab Testing Framework (OSATF) is a Web server for conducting A/B and Multivariate testing. It is designed to allow website developers and other site owners to easily add statistical testing capabilities to any website. The framework includes one of the most popular A/B testing javascript libraries (Google’s Universal Analytics) to provide rich functionality like event tracking, customized domains, etc.


Nelio offers A/B testing facility for all WordPress-based websites. It is open-source A/B testing software that is easy-to-install and activates to your system to create, delete, and monitor your website’s progress and other related experiments.

(Source: Nelio)


  • Nelio is not just a WordPress plugin, but also offers a conversion optimization service.
  • The software allows you to test the alternative versions of WooCommerce projects.
  • It helps you to create catchy headlines that can achieve the highest click-through rates.
  • The Nelio software offers testing new layouts, new designs, and tweaking finer details using CSS experiments to redesign their website.

Google Optimize

Free A/B Testing Tool - Convert as an alternative to Google Optimize

Google Optimize, formerly Google Analytics Content Experiment, has major benefits in its natural integration with Google Analytics. For a free tool, Google Optimize has several features, including:

  • A/B testing
  • Multivariate testing (MVT) 
  • A variety of targeting options, including Javascript and Geo-targeting. 

A/B, A/B/n, MVT, Split URL? Yes.

Server-side and/or client-side? Both.

WYSIWYG editor? Yes.

What type of customer support is available? Resource hub with a community, starter guide, video tutorials, tips, inspiration, etc. No live support, email, or phone number. 

Premium version available? Yes. Google Optimize is available at custom pricing.

GDPR compliant? Yes.


  • All-in-one solution, seamless integration with other Google products
  • Working together with Google Analytics, it provides in-depth reports
  • Super easy to use for a first-timer


Open Source Ab Testing Framework Image

Conductrics is my favorite experimentation platform. I like the CEO, Matt Gershoff, and he’s been an excellent industry mentor and friend to me. He is also one of the most thoughtful leaders in the analytics and optimization space, so you can trust the direction of the platform.

He has also built a really great piece of software that, in many ways, was far ahead of its time and definitely ahead of most of the others on the list.

Power/feature-wise, it’s the best you can get. You can run A/B tests (client or server side), multivariate experiments, and bandit tests, all from a WYSIWYG visual editor (if you want). You don’t need to install a script until you set things live, as you can use the browser extension to edit and preview variants.

Additionally, the tool lets you run predictive targeting to exploit different segments of your audience. Basically their AI/machine learning will discover which combinations of visitor traits respond best to each of your content variants, automatically.

Definitely a tool for true optimizers.


Open Source Ab Testing Framework Image

Optimizely is the biggest brand names on the list, and I have to say, I’ve always loved using the platform. It made javascript, client-side A/B testing world popular. It’s what I learned on, so I’m definitely most comfortable in this interface, even to this day.

Feature-wise, they’ve really improved in the last few years, especially in their enterprise offering. They’ve now got ‘full-stack’ experimentation capabilities, valuable for those who want to run server-side experiments or run tests on IoT, TV, or mobile apps. The also offer unlimited free feature flags and rollouts.

Overall, it’s a robust and powerful enterprise solution that even features a fully fleshed out program management platform (though unfortunately the company doesn’t really maintain support for smaller companies nowadays – it’s really only at tool for large brands).

Their stats engine is based on sequential testing, which is often more efficient, but I do love a bit more flexibility in terms of statistical analysis.

Optimizely is probably in that tier of software where, like IBM, ‘you won’t get fired for buying Optimizely.’


Free A/B Testing Software

Vanity is another free and open-source optimization platform that describes itself as “an Experiment Driven Development framework for Rails.”

If you are not familiar with Rails, it’s a Ruby (programming language) web application development framework.  

Vanity allows users to measure multiple metrics through A/B testing.

Sidebar A/B Test results

A/B, A/B/n, MVT, Split URL? Only A/B testing and code testing.

WYSIWYG editor? No.

Server-side and/or client-side? N/A

What type of customer support is available? None. Only setup instructions in the README. If you have questions, you can ask on StackOverflow.

Premium version available? No.

GDPR compliant? Unclear.


If you are familiar with Rails, and use this as your development platform, having access to this open-source tool that allows you to run small A/B tests is handy. 


However, this software has limited reach. It does not allow you to test full websites, run concurrent tests, or a host of other functions you would expect in a full A/B testing tool.


Cucumber is a combination of open-source and commercial test tools, with over 30 million downloads all across the world. It is also ranked as the number #1 tool for Behavior-Driven Development.

(Source: Cucumber)


  • Cucumber is available for multiplatform programming languages including Node.js, Java, C++, Ruby on Rails, Python, and so on.
  • Cucumber software can work well with third-party API automation tools to make your tests faster and less flaky.
  • Stable version ‎3.1.2 of the software was released on July 13, 2018, and is licensed as per MIT licensing.
  • Reporter plugins are available to produce reports informing what scenarios are passed or failed.


Open Source Ab Testing Framework Image

VWO is similar to Convert in that it’s much more affordable than Optimizely (though VWO has gotten more expensive in recent years), and it has many of the fundamental features you’d want. Of course, you can run A/B tests, split path tests, multivariate tests, and personalization targeting.

You can also use their other CRO features, things like surveys, heatmaps, session recording replays, funnels (and the rest of HotJar’s feature list, basically). If you like all-in-one, VWO is the way to go (they even offer solutions for server-side and mobile testing).

They offer other less-CRO-related tools, as well, like push notifications.

Overall, their CRO-suite is top-notch and feature rich, and I love using VWO. They offer a free 30 day trial, so again, you’ve got nothing to lose from checking it out. VWO is an excellent option for optimizers at all sizes of companies.


The Open Source Ab Testing Framework project brings together a variety of tools and utilities to help you run your tests. Many problems associated with testing have been solved once and for all. Just use the framework for your current project, instead of building everything from scratch.

Similar Posts


No Comment.