Okay, your understanding of "Selenium Server" & "Selenium WebDriver" is wrong.
Selenium Server: This Was extensively used in Selenium RC for performing local and remote tests because of Same Orgin Policy. However, in Selenium WebDriver, there is no need for Selenium Server JAR for performing local tests. Only for remote tests using Grid, it is needed.
And with the other zip package, we have all the classes & methods as libraries for allowing Selenium WebDriver to achieve automation while testing the web pages.