What encryption/encoding is being used here?

That seems to be a custom written encryption algorithm since it does not follow common encryption algorithms for that short of word lengths. I would have to spend some time running complex tests and ysis algorithms to attempt to crack that encryption. It's most likely a simple algorithm that you can crack with the right ysis tools.