We’ve put between the hands of a select group of our customers, a version of Netflix Ice running against their AWS account billing. The idea was to get functional feedback from “real users”.

Context of the test

These users had already been using Teevity cloud costs analytics for several month and we have a direct connection with them.

They did not auto-discover the product, we pushed it at them and gave access to it pre-configured for their AWS accounts, as a SaaS solution but isolated from our existing Teevity SaaS service. So they had in their hands a “very easy to get access to” version of Ice, with functional support provided by us.

This first test didn’t cover all aspects of Ice (AWS Reservations for instance where not covered by the test).

Feedback 1 - Ice is very good for AWS cost debugging

One thing which comes back from all the participants is that Ice is very good at “debugging” your cloud costs. The graphical view it provides enable a deep dive inside the cost and usage data that AWS dumps inside your Programmatic Billing bucket.

[[MORE]]

Feedback 2 - Ice comes with a powerful “factor throughput” concept built-in

Another point that some participants liked very much is that Ice has a “Factor throughput” feature that they would have liked to use. It’s not useable out of the box from the existing code base but it aims at:

  1. showing business metrics along side your cost timeline
  2. factorizing these cloud costs y the business metric to show your ‘cloud cost per unit of [business metric]

Feedback 3 - Frustration for missing features, lack of data from Programmatic Billing or Cost Allocation Reports tagging

The Ice code published on GitHub is quite incomplete (probably because before being published, the code had to be cleaned up from Netflix specifics) and so Ice, as it is on Github today, lacks several features that at least some companies need regarding cloud costs management

  • easy cost sharing between people in the company,
  • historical analysis going before Programmatic Billing which is not retroactive,
  • cost classification beyond AWS tagging which are not retroactive either,
  • access protection,
  • and an API (if you want to integrate your costs in Excel for instance)

AWS Programmatic Billing and Cost Allocation Report are not retroactive

The other thing that caused frustration is not related to Ice but to the way AWS Programmatic Billing and Cost Allocation Report work: they are not retro-active.

You cannot go further in your cost exploration than the initial date of activation of Programmatic Billing (which is why you should activate Programmatic Billing now !)

When Tags are used inside the Cost Allocation Report, they only apply from the time you apply them. You need to quickly get them right, because until you stabilize them for a full month, your cost reports will be hard to read.

Feedback 4 - Some customers don’t love SaaS that much

One of these customers, a Europe-based US army supplier who has strong confidentiality requirements, was very interested in this test because Ice “could run all inside [his] AWS account” (since it was started on his account with a CloudFormation) but was “as easy as SaaS”.

AWS Marketplace vs SaaS

That’s an interesting feedback! And I guess what is true for cost analytics will be true for many other solutions: the “AWS Marketplace” option will probably be a serious challenger to the “SaaS option” in the years to come, especially as the AWS Marketplace matures (and is able to launch full CloudFormations instead of just single EC2 instances).

Conclusion

When used as a Cloud cost debugging tool, the feedback on Ice has been very good.

Ice was thus seen as very complementary to Teevity’s existing feature set and a very good addition for deep “AWS cloud costs analysis”.

This is why, as announced in our previous post, Ice features will make their way into Teevity cloud costs analytics and will complement the existing features which make it easy to share cost data inside the company and to fetch historical cost and usage data.