Training Deep Learning Models on AWS Pt. 1 - Expense Tracking
Set up budget alerts
Before you even start thinking about training a model, I HIGHLY recommend setting up a budget with reporting alerts. This ensures you know exactly how much money is spent and that no instance or storage is accidentally left on for too long.
First, find the AWS budgets page. You can find it quickly by searching “budget” from the AWS Console (after you are signed in).
Once on the Budgets page, click “Create budget.”
Select “Cost budget” as the budget type.
Give your budget a useful name, like the name of the project or model. From here on I will be using one of my existing budgets as an example. In my case, I was training a model named “Bernice” and I had a budget of $5K. I knew I would be training the model at some point in 2022 but didn’t know the exact month, so I just set the budget for the whole year. You can be more specific for the month/day if you have more information.
For the alerts, you have the option to set up relative or absolute thresholds. I set up absolute thresholds for $50, $100, $1000, $2000, $3000, and $4000. I also set up a relative threshold for 90% of my budget. You can also choose between being alerted when the actual value is hit or when the value is forecasted. I chose actual since I set up so many alerts and forecasted was not needed for my situation. For the alert, I provided an email that I check frequently.
After the alert page (click “next”), you have the option to add actions. I did not add any actions (I don’t want my model to be automatically stopped), but if your budget is particularly tight or your EC2 instances have a high hourly rate, adding an automatic stop action is useful. You need the EC2 instance ID to set this up, if it’s not running yet (since you should set up alerts FIRST) then you can add the action later by editing your budget.
Now that our budget alerts are set up, let’s make a nice cost report!
Create cost report
The budget is useful for alerts, but I also want a pretty chart of my costs on a per-type level (e.g., EC2 and EBS storage).
Similar to finding the budget page, find the AWS Cost Explorer by using the search bar on the AWS Console home page. In the Explorer, check the menu on the left-hand side for “Reports”.
There are some default reports, but we want to create our own. Click “Create report”.
Create a “Cost and usage” report.
There are many useful filtering settings for the cost reports. For mine I wanted the see the entire year (Year To Day, YTD) and for my expenses to be grouped by usage type. This groups my EC2 instances together and my EBS storage costs together. I also want to see the cost on a daily basis, but this can also be viewed hourly or monthly. Since my account is using AWS credits, I had to exclude “Credit” from “Charge Type” to see the cost.
We have our budget alerts and our pretty cost report all set up! Good job!