Back to Blog

Google Vertex vs Masterful AI (Part 1)

Image of Jack Lynch
Jack Lynch

Percent top-1 error reductions achieved when moving from Google Vertex to Masterful AI. 


 

As more companies incorporate machine learning into their products, interest has grown in capable AutoML tools that can help engineers improve their models without additional expertise or technical debt.

Vertex AI is a popular AutoML platform which, like Masterful, powers the model-training process to improve performance. But how do the two platforms compare? This is the first in a series of blog posts answering this question. For now, we’ll focus on a simple scenario: a user has a reasonable amount of labeled data and wants to train the best model they can using it.

We evaluated Vertex AI and Masterful on a set of well-known datasets spanning geospatial, medical, and natural imagery. For geospatial imagery, we used the EuroSAT dataset of 64x64 images taken from Sentinel-2 satellite imagery, covering 8 object classes.

eurosat

Sample EuroSAT images.

For medical imagery, we used three datasets of the larger MedMNIST collection: DermaMNIST (dermatoscopy), PathMNIST (colon pathology), and BloodMNIST (blood-cell microscopy).

mednmist

Sample MedMNIST images (from left to right: DermaMNIST, PathMNIST, and BloodMNIST).

For natural imagery, we used the prominent CIFAR10 and CIFAR100 datasets, which consist of 32x32 color images of 10 and 100 object classes, respectively.

cifar10

Sample CIFAR10 images.

When using Masterful, we observe a significant gain in top-1 accuracy over Vertex across the board, as shown below:

Screen Shot 2022-03-22 at 3.16.08 AM

Top-1 accuracies for Google Vertex and Masterful AI on each dataset.

With a standard model architecture (WideResNet-28-10 for the smaller-dimension datasets, ResNet-50 for EuroSAT), Masterful delivers a boost in performance using the same labeled data.

If you’re curious in trying Masterful out for yourself, you can download the pip package and try it out: 

carbon (6)

You can read the docs here. Masterful meta-learns the optimal training techniques and hyperparameters to improve your model, from augmentation choices to batch size and learning rate, so you don't have to spend hours testing and tuning.

Vertex also supports model deployment, which may be important for existing users of the platform. However, it’s fairly simple to continue deploying with Vertex even if you train with Masterful. When you’re finished training a model with Masterful, you can export it to the SavedModel format and upload it to Vertex for deployment, effectively bypassing Vertex’s training process with no additional headaches. If you aren’t already using Vertex, there are a number of other paths to model deployment, ranging from the complex (AWS SageMaker) to the simple (Banana).

In the second post of this series, we’ll examine how to push the Masterful advantage even further by incorporating unlabeled data. In the meantime, join our Slack community!


Related Posts

Hello World

Image of Sam Wookey
Sam Wookey

We're thrilled to announce that the Masterful AutoML platform is now available on the Python...

Read more

10x your Computer Vision development with Masterful Low-Code

Image of Tom Rikert
Tom Rikert

Today we’re making a big leap forward in enabling new applications and insights for every...

Read more