Brain-computer interfaces (BCIs) can significantly aid individuals with severe motor and communication disabilities by enabling the recognition of neural signals for activities like handwriting. This study introduces the first implementation of neural signal inference on a portable device for efficient handwritten character recognition. Using neural spike-event data from a publicly available dataset, researchers classified 31 handwritten characters on an NVIDIA Jetson TX2. To improve model generalization and reduce overfitting, techniques such as random noise injection and time-shifting data augmentation were applied. The approach utilized EfficientNetB0 with neural spikes, achieving a test accuracy of 99.17%, which is a notable improvement over previous models. The EfficientNetB0 demonstrated a Word Error Rate (WER) of 0.96% and a Character Error Rate (CER) of 0.2% during rapid inference. It processed 100 daily life sentences with a character decoding latency of 37.5 milliseconds on the Jetson TX2. These findings confirm the viability of precise, high-speed neural decoding on portable hardware, showcasing the potential of lightweight machine learning models in BCI applications.