In this paper we consider the problem of computing decentralized control policies for stochastic systems with finite state and action spaces. Synthesis of optimal decentralized policies for such problems is known to be NP-hard. Here we focus on methods for efficiently computing meaningful suboptimal decentralized control policies. The algorithms we present here are based on approximation of optimal Q-functions. We show that the performance loss associated with choosing decentralized policies with respect to an approximate Q-function is related to the approximation error. We demonstrate the methods developed in this paper with an example of load balancing in a queuing network.