Machine learning achieved phenomenal empirical successes. However, these results are based on a tremendous computation power and require hundreds of thousands of GPU hours or thousands of years of human-equivalent time to train. Such an enormous need of computational resources severely limits the applicability of ML. Algorithmic methods are a promising direction for scalable machine learning to reduce communication costs in distributed and federated learning while maintaining training accuracy, compressing deep neural networks by pruning “insignificant” neurons, improving selective plasticity methods in continual learning. Our recent theoretical and empirical results showing high accuracy with pre-training compression for specific classes of activation functions, and we intend to investigate the viability of the extension of these techniques. We will explore approximate methods for federated learning and analytics that balance accuracy with other critical constraints, such as inference costs, privacy, or fairness delivering principled methods for heterogeneous distributed networks that can flexibly trade off between a number of competing goals. We will apply algorithmic techniques to reduce model size of reinforcement learning and hence reduce the need for samples and training time.