Sasha Madden Ebersole
Aspiring ML engineer with experience in frontend and full-stack web development.
sasha.m.ebersole@gmail.comGitHubLinkedInAbout Sasha
I am transitioning into machine learning engineering after working as a frontend developer at DataMixMaster and a full-stack developer at Groenewold IT Solutions.
Outside of work I like to read about late imperial Chinese history, train Muay Thai, write poetry and short fiction, and draw.Projects
Dots and Circles is my first non-tutorial machine learning project. It is an app for optical character recognition of Manchu-language text.
Problem
The Manchu script is written top-to-bottom, left-to-right. All letters within a word are connected to each other, and the shape of a letter changes based on its position within a word. This makes it quite difficult to break a word down into individual letters. Additionally, this problem was to be tackled without the help of a framework like PyTorch, since I wanted to get a better sence of how neural networks work under the hood.
Approach
Training data was synthetically generated by rendering words in various fonts. In each word, each row of pixels was labelled with the corresponding unicode character. Pairs of neural networks predict which characters correspond to lines in an image. Each pair is specialized for a particular style of Manchu text. An RNN determines which pair to use for a given image by reading the image line by line. The final output is then collapsed from an array of tokens into a single word.
Next Steps
Manually labelling characters was time-intensive, resulting in a small training set. The lack of training data meant that the model was unable to effectively generalize to other words, resulting in low accuracy. In the future, I aim to rebuild this using an encoder/decoder model trained on entire words, allowing it to use existing datasets.
Raschka LLM
GitHub repoA GPT-style LLM created by working through Sebastian Raschka's book 'How to Build a Large Language Model (From Scratch)'
Problem
Prior to reading this book, I had no idea how LLMs worked under the hood. Additionally, I had no experience using PyTorch. By working through Raschka's book, I hoped to gain a deeper understanding of LLMs and deep learning in general.
Approach
Working chapter by chapter, I implemented the LLM with PyTorch. Each chapter involved iterating upon some component of the GPT model, from a basic example to a full implementation. In the course of this process, I implemented token and position embedding, multi-head attention, layer normalization and shortcut layers, transformer blocks. Once this was done, the LLM was at first trained on a short story learning to predict the next word in a sequence. Finally, GPT-2's public weights were loaded and then fine-tuned for two tasks: spam classification and responding to plain language prompts.
Next Steps
Over the course of this book I learned a lot about model architecture and training, but the two big takeaways were the utility of transformers and an understanding of fine-tuning existing models. There are two projects I would love to apply these to: first, using transformers to create a model capable of machine translation between English and either Classical Chinese or Manchu. Second, combining RAG with fine-tuning to produce a model that can act as a reliable interface between a user and the Qing Vertable Records.