Google Vertex vs Masterful AI (Part 1)

Percent top-1 error reductions achieved when moving from Google Vertex to Masterful AI. 

As more companies incorporate machine learning into their products, interest has grown in capable AutoML tools that can help engineers improve their models without additional expertise or technical debt.

Vertex AI is a popular AutoML platform which, like Masterful, powers the model-training process to improve performance. But how do the two platforms compare? This is the first in a series of blog posts answering this question. For now, we’ll focus on a simple scenario: a user has a reasonable amount of labeled data and wants to train the best model they can using it.

We evaluated Vertex AI and Masterful on a set of well-known datasets spanning geospatial, medical, and natural imagery. For geospatial imagery, we used the EuroSAT dataset of 64x64 images taken from Sentinel-2 satellite imagery, covering 8 object classes.

Sample EuroSAT images.

For medical imagery, we used three datasets of the larger MedMNIST collection: DermaMNIST (dermatoscopy), PathMNIST (colon pathology), and BloodMNIST (blood-cell microscopy).

Sample MedMNIST images (from left to right: DermaMNIST, PathMNIST, and BloodMNIST).

For natural imagery, we used the prominent CIFAR10 and CIFAR100 datasets, which consist of 32x32 color images of 10 and 100 object classes, respectively.

Sample CIFAR10 images.

When using Masterful, we observe a significant gain in top-1 accuracy over Vertex across the board, as shown below:

Top-1 accuracies for Google Vertex and Masterful AI on each dataset.

With a standard model architecture (WideResNet-28-10 for the smaller-dimension datasets, ResNet-50 for EuroSAT), Masterful delivers a boost in performance using the same labeled data.

If you’re curious in trying Masterful out for yourself, you can download the pip package and try it out: 

You can read the docs here. Masterful meta-learns the optimal training techniques and hyperparameters to improve your model, from augmentation choices to batch size and learning rate, so you don't have to spend hours testing and tuning.

Vertex also supports model deployment, which may be important for existing users of the platform. However, it’s fairly simple to continue deploying with Vertex even if you train with Masterful. When you’re finished training a model with Masterful, you can export it to the SavedModel format and upload it to Vertex for deployment, effectively bypassing Vertex’s training process with no additional headaches. If you aren’t already using Vertex, there are a number of other paths to model deployment, ranging from the complex (AWS SageMaker) to the simple (Banana).

In the second post of this series, we’ll examine how to push the Masterful advantage even further by incorporating unlabeled data. In the meantime, join our Slack community!