OpenGradient Modelthon Jan-Feb 2025 Detailed Information

Last Updated: February 21, 2025 6:00 PM EST

BTC Track Leaderboard

Rank	Model Name	MZTAE Score
1	log-transform-btcusdt	0.360571
2	LightGBM_BTCUSDT_1Hour_Return	0.374208
3	seabiscuit_BTC	0.375817
4	XGBoost_regression_mode_btcusdt	0.376922
5	og_modelthon_xgb_btcusdt_1hour_return	0.385919

ETH Track Leaderboard

Rank	Model Name	MZTAE Score
1	log-transform02-ETHUSDT	0.494102
2	og-ethusdt-1h-return-xgb-model	0.500649
3	simple_ETH_3_1	0.500834
4	knn-1hour-return-ethusdt	0.512481
5	OG_Modelthon_LightGBM_ETH_Price	0.568782

Note: MZTAE (Mean Z-tanh Absolute Error) is the scoring metric used for the competition. Lower scores indicate better performance.

Important Dates

• Registration Deadline: Feb 13
• Model Submission Deadline: Feb 18
• Spot Model Live Data Run: Feb 18 to Mar 11
• Winners for all competitions announced: Mar 11-13

Building a Model

Data

Required training data for ETH and BTC Spot Forecast tracks are found in the OG Modelthon Github repo. Data descriptions are in the repo's readme.

For the Freestyle track, the provided ETH or BTC data can be used, or any data the competitor can provide for themselves and may legally use. In the Freestyle track the competitor is not required to submit their training data, but a sample input must be provided to demonstrate that model will produce an inference. Additionally the data must be sufficiently described in the Model Hub description to receive full points. Keep in mind that a Freestyle may be deemed more useful if the data used is readily obtainable by other users of OpenGradient.

Using ONNX

You may use any framework to train your model - linear regression, lasso, ridge, neural network, etc. - but your model must be converted to ONNX to run inference on OpenGradient.

What is ONNX? "ONNX is an open format built to represent machine learning models." It allows AI/ML models trained using various architectures and frameworks to run inference in a system-agnostic way. If you train a model using Pytorch or sklearn, you can convert you model to ONNX format so it can run in this open format. You can even create models directly from scratch by building an ONNX graph. ONNX models uploaded to Model Hub can run inferences on OpenGradient.

There are various ways to create ONNX models from scratch and packages to convert from various frameworks. See the official ONNX page for detailed information on conversion.

Our OG Modelthon Github repo provides some guides and examples of properly building an ONNX model for inference including preprocessing (if your ML model requires some feature engineering before direct inference you will need to implement that feature engineering in ONNX). There are also many resources and examples easily searchable online.

Rules

In order to be considered for the competition, a model must be uploaded to OpenGradient Model Hub and a Model Submission Form filled out properly and submitted. A thorough description should be provided in Model Hub, and is required to receive full scoring points.

In order to be considered for the ETH and BTC Spot Forecasting tracks, models must not only be uploaded in ONNX format but must be confirmed to work with the structure of input data as confirmed via test code and infer using the live data input which is of the same format as that tested in the above code. Note that the input name of the data for the model is 'candles', which is what the input name of your ONNX model must be. Guides and links for ONNX conversion found below.

Only one model may compete per competitor per track. If multiple models are submitted by the same competitor to a single track then only the last submission will be considered.

Competitors may only win the competition for a single track. If a competitor wins multiple tracks they will only be recognized and receive winnings for one track.

Models must be original and your own work, i.e. do not simply submit a CID for another person's model or upload a model made by someone else.

Winning models and competitors are subject to scrutiny to ensure that rules and laws have been followed.

Scoring criteria

ETH and BTC Spot Forecast Tracks

Spot forecast model inferences (forecasts) will be run hourly during the Spot Model Live Data Run period (see Important Dates). Each hour the model's inference forecast for the period will be retrieved and compared to the true calculated value.

• Inference forecast retrieved
• ZTAE (see below for definition) calculated for this period
• MZTAE (see below) calculated for all periods up to and including current
• All models in the competition are given a MZTAE quantile where 100 is the lowest (best) MZTAE and 0 is the highest (worst) MZTAE

Final Scoring

• Performance Score = 0.85 $\times$ MZTAE quantile for entire Spot Model Live Data Run
• Description Score given by judges will be a number between 0 and 15, see a suggested description template in the Modelthon repo
• Total Final Score = Performance Score + Description Score, a number between 0 and 100

MZTAE

The Spot Forecast will use MZTAE as the scoring metric because the target is return and simply calculating MAE on return does a poor job of rewarding what is actually very good performance. If a model forecasts a very high positive return and the true value is indeed some high value, due to high dispersion of extreme values the true value may be quite "far" from the forecast as measured by MAE despite the forecast being very useful in practicality. MZTAE rewards forecasting extreme values in the correct direction (up or down) while maintaing close to identity error for middling (near zero) values. ZTAE and MZTAE is as defined below:

ZTAE ${\epsilon }^{\zeta }$: $Z$ tanh absolute error

$$\zeta (x)=\text{tanh}(\frac{x-\hat{\mu }}{\hat{\sigma }})$$$${\epsilon }^{\zeta }=|\zeta (\hat{x})-\zeta (x)|$$

where $x$ is the true value and $\hat{x}$ is the forecast

MZTAE $\overline{{\epsilon }^{\zeta }}$: mean ZTAE

$$\overline{{\epsilon }^{\zeta }}=\frac{1}{n}\sum _{i=1}^n{\epsilon }_i^{\zeta }$$

Freestyle Track

Judges selected by OpenGradient & Allora will review model descriptions and results. Judges will also consider the usefulness of the model & its contribution to the Web3 space. The model description must include a sample input to confirm that the model will produce an inference on OpenGradient. Judges will award a score between 0 and 100.

Uploading and Testing Inference

Your ONNX model should be uploaded to Model Hub which can be done directly on the site or with the Python SDK.

You should install the OpenGradient Python SDK, and confirm that your uploaded will run using the SDK. For spot forecasting models, be sure the model runs using specified input format using sample test code found in the OG Modelthon repo.

Remember, for Spot Forecast tracks the input name must be 'candles' with a calculable input shape (1,81).

Don't forget to add a thorough description for your model in Model Hub. There is a suggested model description format in the Modelthon repo.

AFTER YOU HAVE BUILT, UPLOADED, AND TESTED YOUR MODEL you must fill out the Model Submission Form in order for you model to be entered into the contest.

After submitting your model and completing the submission form before the submission deadline (Feb 18 00:00 UTC), you can sit back and relax. Spot Forecast track models will be run on live data for properly submitted, functional models (information about the Modelthon Dashboard to track model performance will be released prior to the Spot Model Live Data Run), and Freestyle track models will be judged.