An Introduction to Mathematical Cryptography
Jeffrey Hoffstein, Jill Pipher and Joseph H. Silverman
This book focuses on public key cryptography and no prerequisites in higher mathematics are
needed, which makes it a great first book in cryptography.