Auto PreTrain
🚂
AI & ML interests
None defined yet.
Recent Activity
FlameF0X
updated
a
Space
10 days ago
FlameF0X
updated
a
model
11 days ago
FlameF0X
published
a
model
11 days ago
Post
4099
I am very sad to say that the budget in creating of SnowflakeCore-G1 1b and 7b MoE models ran out and I can't pre-train them anymore.
-
- 7 replies
Post
545
the training for SnowflakeCore-G1-1B and 7B would be retaken because now I implemented DeepSpeed and management to use two gpus.
Post
274
The development of SnowflakeCore-G1-7B-MoE it getting delay. In the mean time I am working on SnowflakeCore-G1-1B-MoE witch would be a pre-train chatbot.
-
- 1 reply
Post
2957
The development of SnowflakeCore-G1-7B-MoE. I can't say when it would be publish yet because it's big and it requires a lot of computational power.
-
- 1 reply
Post
292
I just finished the benchmarks for https://huggingface.co/FlameF0X/SnowflakeCore-G1-Tiny and https://huggingface.co/FlameF0X/SnowflakeCore-G1-Tiny2 in comparation with
openai-community/gpt2 .
Post
314
Hello! Important announcement, I will rename SnowflakeCore-G1-Medium to SnowflakeCore-G1-Tiny2 because it's going to have the same parameters as the Tiny version, but this one is trained on more data.
- 1 reply
Post
746
Currently working on SnowflakeCore-G1-Medium. [Updated loss cruve]
-
- 3 replies
Post
156
Hello there world! I am happy to announce that you now can fine-tune https://huggingface.co/FlameF0X/SnowflakeCore-G1-Tiny , the code for that is in the model card.
I aslo lost the training log 😐
I aslo lost the training log 😐
Post
1207
Hello! I am sad to say but fine-tuning https://huggingface.co/FlameF0X/SnowflakeCore-G1-Tiny is complicated and the instruct version would need to wait some time.
-
- 2 replies
Post
230
SnowflakeCore-G1-Tiny has landed on Hugging Face! 🚀. Give it a try and let me know what you think: https://huggingface.co/FlameF0X/SnowflakeCore-G1-Tiny.
Post
257
SnowflakeCore-G1 Update:
Got it running and training! Context window is currently set to 2048 tokens.
Training is active and stable. Will share results once I have some metrics to report.
Got it running and training! Context window is currently set to 2048 tokens.
Training is active and stable. Will share results once I have some metrics to report.
-
- 2 replies
Post
1938
SnowflakeCore-G1 development update: We're building a 24-layer transformer with 32K context and 1024 embedding dimensions - pretty ambitious! Even running at batch_size=1 with heavy gradient accumulation, we're hitting memory walls at 300GB RAM. Scaling up to ~1TB will take some time, but the architecture is looking promising. Thanks for following along with the journey! 😅
-
- 1 reply
Post
1153
Hello there!
I just find out that all the SnowflakeCore-G0 series are Mask Language Models instead of LLM's.
The development of SnowflakeCore-G0-Releas-3 would be delayed even more.
Edit: I officially end the development of SnowflakeCore-G0 and start the development of SnowflakeCore-G1 what SHOULD be the text generator.
Edit-2: After some evaluation of the code, the models are actual Text Generator. So the development of G0 will continue.
I just find out that all the SnowflakeCore-G0 series are Mask Language Models instead of LLM's.
The development of SnowflakeCore-G0-Releas-3 would be delayed even more.
Edit: I officially end the development of SnowflakeCore-G0 and start the development of SnowflakeCore-G1 what SHOULD be the text generator.
Edit-2: After some evaluation of the code, the models are actual Text Generator. So the development of G0 will continue.
Post
1376
Hi everyone!
The release of https://huggingface.co/FlameF0X/SnowflakeCore-G0-Release-3-1B is currently delayed due to hardware limitations—I'm currently lacking the compute resources needed to complete training. I'm exploring options and will keep you updated on any progress.
Thank you for your patience and support!
The release of https://huggingface.co/FlameF0X/SnowflakeCore-G0-Release-3-1B is currently delayed due to hardware limitations—I'm currently lacking the compute resources needed to complete training. I'm exploring options and will keep you updated on any progress.
Thank you for your patience and support!
Post
213
Hi there! I'm currently developing https://huggingface.co/FlameF0X/SnowflakeCore-G0-Release-3-1B and have estimated the model size at approximately 1.06B parameters. However, this value is only an estimate, and I don't have the exact parameter count.
Post
1981
I realised a small documentation on how to make your own LM architecture called [LM-From-Scratch](https://github.com/FlameF0X/LM-From-Scratch)