Should I have separate builds for development and release?

To me, it's clearly worth building with debug symbols, profiling etc. during development.
Is it worth having a separate build for production with no debug symbols and aggressive compiler optimization?

I'm unsure because it seems like it would make debugging easier if the production build was the same as the one used in development, but then you're possibly losing out on some performance caused by optimization and making your program bigger than it needs to be by including debug symbols…