In a computer database, the database engine is the software that does the real work of sorting the information, finding specific data that you request, and so on. The term used to refer to a separate piece of software that ran on a central computer (in this case, it is more or less synonymous with the term "back-end"). Widely used database engines include Oracle, DB2, and Sybase. Separate front-end software running on your own computer lets you tell the database engine what to do (how to sort the data, what data to find), and displays the results of your commands.
If you're talking about database software that runs entirely on one computer, the term database engine is sometimes used to describe that part of the software responsible for manipulating the data, as distinct from the parts of the software that you interact with on the screen.