This book covers in eight chapters both theoretical and practical knowledge in computing on distributed memory systems. It extracts the main ideas and principles of parallel algorithms developed over the last few decades. The theoretical part presents a rigorous, yet accessible, treatment of theoretical models of parallel computation on two classical examples, namely PRAM and the sorting network. In several chapters, the reader is introduced to fundamental results and common problems in scheduling. Advanced topics cover crucial divisible load scheduling and steady-state scheduling, which are applicable to the master-worker application model. The reader will find important topology and performance discussions about different network models. Subsequent chapters use this knowledge to build parallel algorithms on two popular abstractions: ring and grid logical topologies. But the authors perfectly explain not only homogeneous models (which are everyday problems on clusters of identical nodes) but also load balancing on heterogeneous platforms (connecting different clusters or many different workstations). This book can serve as a very good teaching book or a source of useful material for graduate students and researchers in parallel distributed memory architectures. It contains many schemes, diagrams and pictures for better understanding, including many practical examples, case studies and exercises.