Machine Learning: Sudoku Benchmark - Precision policy

Go back to list

Context

For fast computations, Floating Point precision is a high factor of performance. From 16 to 128 bits, the precision plays with the accuracy of training and predictions. Basically, a higher precision also involves slower computations. This is a rule of thumb and specialized hardware such as GPU or TPU may act more effiiently with specific data type. For instance, most of the GPUs generaly perform well with float32 while NPU prefer float16.

Observaton: What is the optimal dtype for a Nvidia Tesla V100 ?

We run the same scenario using the different datastrure types

{ "configuration": { "chart": { "type": "spline", "polar": false, "zoomType": "", "options3d": {}, "height": 500, "width": null, "margin": null, "inverted": false }, "credits": { "enabled": false }, "title": { "text": "" }, "colorAxis": null, "subtitle": { "text": "" }, "xAxis": { "title": { "text": "Precision policy" }, "categories": [ "default", "float16", "float32", "float64", "mixed_float16" ] }, "yAxis": [ { "gridLineWidth": 0, "title": { "text": "Inference score", "style": { "color": "#4BAF50", "font-size": "20px" } }, "opposite": true, "floor": 0, "ceiling": 1, "softMin": 0, "softMax": 1, "labels": { "style": { "color": "#4BAF50" } } }, { "gridLineWidth": 0, "title": { "text": "Loss", "style": { "color": "#8A5CA0", "font-size": "20px" } }, "opposite": true, "floor": 0, "softMin": 0, "labels": { "style": { "color": "#8A5CA0" } } }, { "title": { "text": "Train speed", "style": { "color": "#333333", "font-size": "20px" } }, "floor": 0, "softMin": 0, "labels": { "style": { "color": "#333333" } } } ], "zAxis": { "title": { "text": "" } }, "plotOptions": { "series": { "dataLabels": { "enabled": false, "format": "{series.name}", "distance": 30, "align": "left", "inside": true, "allowOverlap": false, "style": { "fontSize": "17px" } }, "showInLegend": null, "turboThreshold": 1000, "stacking": "", "groupPadding": 0, "centerInCategory": false } }, "rangeSelector": { "enabled": false }, "legend": { "enabled": true, "align": "center", "verticalAlign": "bottom", "layout": "horizontal", "width": null, "margin": 12, "reversed": false }, "series": [ { "name": "Inference score", "data": [ 0.9986, 0.184, 0.997, 0.995, 0.999 ], "lineWidth": 5, "color": "#4BAF50", "marker": { "enabled": 0 } }, { "name": "Loss", "data": [ 0.30791865587234496, null, 0.30466726422309875, 0.304027570271492, 0.3051877021789551 ], "yAxis": 1, "lineWidth": 5, "color": "#8A5CA0", "marker": { "radius": 5 } }, { "name": "Train speed", "data": [ 38571.154687644594, 28977.61459126759, 38430.63132891802, 19324.81119325465, 23229.18672558721 ], "yAxis": 2, "lineWidth": 5, "color": "#333333", "marker": { "enabled": 0 } } ], "drilldown": {}, "tooltip": { "enabled": true, "useHTML": false, "headerFormat": "", "pointFormat": "<span style=\"color:{series.color}\">{series.name}</span>: <b>{point.y:.2f}</b><br/>", "footerFormat": "", "shared": true, "outside": false, "valueDecimals": null, "split": false }, "annotations": null }, "hc_type": "chart", "id": "139902536539664" }

Assertions

  • The defaut policy is the as adpated as the float32
  • Float16 isn't able provide a Loss value and cannot learn how to play Sudoku.
  • Float64 and mixed float16 are slower