AI Development Hardware Metrics and Monitoring (RTX Spark)

Nerd Corner Programming
, , ,
1

Hello nerd corner! We are going to be getting in Nvidia’s brand new RTX Spark laptops soon and doing a ton of AI/ML testing on them. If you know anything about the hardware requirements, metrics, and preferences for this kind of work, we’d love to hear a few things from you as we work on this video:

What hardware metrics do you care about most for your AI/ML work and why?

What sort of performance metrics do you monitor during model training and/or inference and why? Temperatures, clock speeds, etc.

Are the test cases we’re using sufficient to gauge this information? (Sam or Khayam will be posting these down below)

Is there anything else we should be looking out for during testing?

Any and all information would be very helpful as we go into this launch. Thank you very much for your input.

2

Hello guys,
So far we are planning to do AI inferencing and nothing down common metrics such as tokens per second, TOPS, TTFT, thermals. Alongside that we will test out general usefulness of the largest model we can fit in each device depending on the memory allocation. Further than that we are planning to train a model locally on each device, comparing the time that it takes and metrics such as thermals, throttling, and power consumption.