XCode 4 is incredibly slow

XCode 4 has always been incredibly slow for me. The first release, 4.0, was especially bad. But that was just a .0 version. The next release 4.1 is much better, but it has also severe drawbacks, concerning performance. Everytime I start it, and not even do much with it, my system gets incredibly slow. That is on both a C2D 2.8 GHz MBP and also on a quad-core i7 MBP. Both machines come with 4 GB of RAM, and after firing XCode up and loading a large project, still at least 500 MB of it remains free. However, speed is abysmal. I just found the tool vmmap in OS X, and it gives me this output:

==== Summary for process 32136
ReadOnly portion of Libraries: Total=265.8M resident=114.4M(43%) swapped_out_or_unallocated=151.5M(57%)
Writable regions: Total=16.2G written=149.6M(1%) resident=360.9M(2%) swapped_out=6156K(0%) unallocated=15.9G(98%)
REGION TYPE                      VIRTUAL
===========                      =======
CG backing stores                  19.4M
CG image                            268K
CG raster data                     2840K
CG shared images                   3472K
CoreAnimation                       180K
CoreGraphics                         16K
CoreImage                           108K
CoreServices                       1704K
IOKit                              61.2M
MALLOC                            337.4M        see MALLOC ZONE table below
MALLOC (reserved)                  15.6G        reserved VM address space (unallocated)
MALLOC freed, no zone              30.5M
MALLOC guard page                    64K
MALLOC metadata                   128.8M
Memory tag=240                        4K
Memory tag=242                       12K
Memory tag=243                        4K
Memory tag=249                      156K
Memory tag=251                       64K
OpenCL                               60K
OpenGL GLSL                        1372K
OpenGL GLSL (reserved)              128K        reserved VM address space (unallocated)
SQLite page cache                  14.6M
STACK GUARD                        56.1M
Stack                              19.7M
VM_ALLOCATE                        16.1M
__CI_BITMAP                          80K
__DATA                             33.9M
__IMAGE                            1256K
__LINKEDIT                         59.5M
__TEXT                            206.4M
__UNICODE                           544K
mapped file                        72.9M
shared memory                      13.6M
===========                      =======
TOTAL                              16.7G
TOTAL, minus reserved VM space      1.1G

So the virtual memory space that XCode takes is more than 16 GB! The actual memory taken is "only" 1.1 GB, which is still huge, but my Emacs also takes 500 MB with tons of C++, Python and LaTeX buffers open.
The question is: can the unallocated, but reserved 16 GB address space degrade the performance? I have too little knowledge of the workings of virtual memory on Intel CPUs under OS X. But this value seems incredibly huge.

Update: I have asked a question on Stackoverflow, and have gotten some useful answers. What did help was removing my build/ folder from the git. Accidentally, a colleague checked in four files in the build/ folder. This made Xcode very slow, since it was checking the git status during compilation all the time. Still, Xcode 4 is much slower than Xcode 3 after this. So I also upgraded our machines to have at least 8 GB of RAM. This was definitely much of an improvement. It seems that development machines using Xcode 4 should have 8 GB RAM minimum. The more, the better...